Monetization system for images

ABSTRACT

A system that automatically matches images to items like products or brands and augments those images with referral links to web pages associated with those items. Image analysis to determine matching items may for example recognize text within an image, decode barcodes or QR codes in an image, and identify objects within an image. Links may be inserted into a message containing an image, such as a text message or email, at a messaging gateway without any action on the part of the sender (or receiver) of a message. Senders or receivers may opt out of or into the link insertion service. When the recipient of a message containing a matched image clicks an inserted link and performs a transaction, a referral credit may be given to the message sender, the message recipient, or a communication intermediary.

This application is a continuation-in-part of U.S. Utility patentapplication Ser. No. 16/520,209 filed 23 Jul. 2019, which is acontinuation-in-part of U.S. Utility patent application Ser. No.15/826,585 filed 29 Nov. 2017, issued as U.S. Pat. No. 10,402,845, whichis a continuation of U.S. Utility patent application Ser. No. 15/706,637filed 15 Sep. 2017, issued as U.S. Pat. No. 10,169,770, which is acontinuation-in-part of U.S. Utility patent application Ser. No.15/483,791 filed 10 Apr. 2017, issued as U.S. Pat. No. 10,229,427, thespecifications of which are hereby incorporated herein by reference.

BACKGROUND OF THE INVENTION Field of the Invention

One or more embodiments of the invention are related to the fields ofdata processing, electronic communication systems, digital advertising,referral reward systems for example as related to electronic commerce.More particularly, but not by way of limitation, one or more embodimentsof the invention enable a monetization system for images, for examplethat utilizes at least one computer and at least one applicationspecifically programmed to generate trackable referrals based on imagesand track the referrals to monetize the images.

Description of the Related Art

Several technology platforms exist for digital promotion of advertisers,brands, products, and services. However, these existing platforms failto address digital word-of-mouth promotion, or peer-to-peer digitalcommunications, where one consumer or user promotes or mentions a brand,product, merchant or service to one or more other users. Existingplatforms include affiliate networks, social marketing, referralmarketing, and influencer marketing. None of these existing platformsadequately address digital word-of-mouth promotion.

Affiliate networks are designed for professional content creators anddigital publishers. These networks are not appropriate for digitalword-of-mouth promotion, because consumers are generally unable orunwilling to participate in the potentially painstaking processesrequired to join an affiliate network, and to perform the steps involvedin utilizing an affiliate network.

Social marketing platforms are typically limited to display ofadvertisements on social networks. As such these platforms are notactually “social” since they do not directly involve communicationsbetween consumers; instead ads simply appear adjacent to socialinteractions, sometimes utilizing optimization software, for example tomake the ads contextually relevant to the social interaction.

Referral marketing platforms generally involve customized, one-offcampaigns on behalf of an individual advertiser. These platforms are notbuilt as a platform for digital word-of-mouth promotion across multipleproducts, services, or brands.

Influencer marketing platforms enlist influencers with large audiencesfor one-off campaigns. These platforms do not address true digitalpeer-to-peer word-of-mouth referrals among family and friends, forexample.

There are no known platforms that provide practical and effectivedigital word-of-mouth referral capabilities in digital communications.There are no known platforms that are unobtrusive, automatic, simple,easy to use, intuitive, and that fit naturally within a peer-to-peersocial dialogue. There are no known platforms that apply to essentiallyall prominent digital channels that people use to communicate withfriends, family, and colleagues, such as social media, messagingapplications, email, and SMS. There are no known platforms that easilyenable people to embed trackable referral links within their digitalcommunications with peers, provide rewards to the referrers, includingcash-based incentives, and that provide a broad range of coverage ofthings the user recommends. There are no known platforms that integratenaturally within social dialogues and are helpful to the recipient,transparent, and non-promotional.

Some existing systems facilitate insertion of referral links intospecific documents, such as blogs or web pages. These systems may forexample be tools provided by affiliate networks, or plugins for webpublishing tools such as WordPress®. A significant limitation of thesesystems is that they are coupled to specific applications or use cases.There are no known systems that integrate referral generation intogeneral-purpose user input methods, such that the referral generationcapability can be used across multiple applications or use cases.

There are also no known systems that provide a transparent method ofdetecting referrals within message content, inserting trackable referrallinks into communications with no action required on the part of thesending or receiving users, that protects the privacy of the sending andreceiving users, and that rewards message intermediaries for insertingrelevant referral links into messages.

There are also no known systems that analyze images to determineappropriate referral links associated with the images, and that trackusage of these referral links to monetize the images.

For at least the limitations described above there is a need for amonetization system for images that generates trackable referrals basedon image analysis, and tracks referrals in a manner that is universalfor all methods of communication between a referring user and areceiving user.

BRIEF SUMMARY OF THE INVENTION

One or more embodiments described in the specification are related to amonetization system for images. Embodiments of the invention may analyzeimages to enable communication intermediaries or users to integratereferrals to participating merchants into communications; recipients ofthese referrals may digitally access the referred merchant, and thesystem may provide a credit to the referrer or a communicationsintermediary if a recipient completes a transaction with the referredmerchant. Referral generation may be integrated into a mobile deviceoperating system or application or may be integrated within a server ormessaging gateway through which messages are conveyed, so that thereferral capability may be used across multiple applications and formultiple use cases. A communications intermediary may be for example: aprovider or operator of a communications channel or service; a provideror operator of a communications link or service anywhere in the path ofa message between a sender or receiver; or a provider or operator of amessaging gateway or platform that may receive, store, forward, route,convey, direct, resend, transform, format, analyze, combine, filter,translate, edit, post, display, or otherwise handle any communication ormessage.

One or more embodiments of the invention include a database of one ormore of participating merchants, a referral matcher that generatesreferrals to these merchants, and a referral tracker that tracks when areferral results in a transaction. The referral matcher may receiveinput from a referring user, for example as part of a communicationgenerated by this referring user; it may then analyze this input toidentify one or more products, services, merchants, brands, orpromotions in the database that match the input. The referral matchermay then automatically generate a referral link and may insert thisreferral link into the communication. The referral matcher mayautomatically insert the referral link into the communication, or thereferral matcher may prompt the referring user or receiving user toagree to the insertion of the referral link and may then insert thereferral link into the communication. The referral link may alsoincorporate an identifier of the referring user or the communicationsintermediary, so that the referral may be tracked. A recipient, orreceiving user, of the communication, may use the referral link toaccess a website or “site” or commerce site associated with the referredmerchant. If the recipient performs a task, for example a transaction onthis commerce site, the referral tracker generates a credit for thesuccessful referral. This credit may be collected from the referredmerchant. All, none or a portion of the credit may be remitted to orotherwise associated with the referring user or communicationsintermediary.

One or more embodiments of the system may analyze input associated withany type of digital communication, including for example, withoutlimitation, a text message, an email message, a communication via asocial media site, a posting on a message board, a communication via amessaging app, a Twitter® message, a Facebook® post, a Snapchat®message, a voice message, a voice signal that is converted to text, avideo message, a picture message, a communication via Facebook®, acommunication via social media, a communication via a shopping site, acommunication via a message board, a posting to a product reviewservice, a digital communication, a comment posted to a digital mediaservice, and a communication via a messaging application.

In one or more embodiments, a communication from a referring user mayinclude or may be generated by sharing of information via a share buttonor other sharing capability. A share button may for example be a nativeshare button in a web browser or mobile operating system (such as iOS®or Android®), or it may be a button or an icon on a website or on anyother document or application.

In one or more embodiments, a referral tracker may generate a feedbackmessage to the communications intermediary or referring user thatindicates that a recipient has executed a transaction as a result of thereferral. In addition to or instead of feedback, the referral maygenerate a credit to the communications intermediary or referring user,which may for example be a monetary payment or another type of reward.In one or more embodiments, the referral credit may be provided to agroup of users. In one or more embodiments, the referral credit may beshared with the recipient as a result of executing the referredtransaction. In one or more embodiments, the referral credit may be adonation to another organization, for example a non-profit or charitableorganization, made for example on behalf of the referring user.

Input into a referral matcher may be obtained from any type of physicaldevice, application, utility, or software program, or combinationthereof, and it may be in any format. For example, in one or moreembodiments the referral matcher may be coupled to a physical or virtualkeyboard used by the referring user, and it may accept and analyzekeystrokes obtained from this keyboard. In one or more embodiments, thereferral matcher may be coupled to an image capture device, such as forexample a camera on a mobile device; the referral matcher may analyzeimages to determine matching products, services, merchants, brands, orpromotions. Images may be for example, without limitation, images ofbarcodes (linear and 2D codes such as QR codes), or images of a product.In one or more embodiments, the referral matcher may be coupled to anaudio input device, such as for example a microphone on a mobile device;the referral matcher may perform voice recognition and analysis todetermine matching products, services, merchants, brands, or promotionsfor example associated with the inferred reference in the communication.In one or more embodiments, the referral matcher may be coupled tosoftware which analyzes content such as voice, images or video; thereferral matcher may analyze this content in any manner to determinematching products, services, merchants, brands, or promotions forexample associated with the inferred reference in the communication

In one or more embodiments, the referral matcher may be coupled to amessaging service, communications platform, network, gateway, server orother application which conveys, transmits, routes, directs, receives,stores, forwards, resends, transforms, analyzes, combines, filters,translates, formats, edits, posts, displays, or otherwise handlesmessages or communications of any type or types. A message may containfor example, without limitation, any type or types of content such asany combination of text, image, voice, audio, video, animations, code,software, links, URLs, website addresses, attachments. The referralmatcher may perform analysis of content that is being transmitted todetect matching products, services, merchants, brands, or promotions forexample associated with the inferred reference in the communication.

In one or more embodiments, the referral matcher may be coupled to asocial media service, digital media service or messaging application,such as for example, without limitation, Facebook®, WhatsApp®, Twitter®,Yelp® or Snapchat®; the referral matcher may perform analysis of contentthat has been input to such an application to determine matchingproducts, services, merchants, brands, or promotions for exampleassociated with the inferred reference in the communication.

In one or more embodiments, the referral matcher may be coupled to therecipient's device, an application used by the recipient, a messaginggateway or network through which messages are conveyed to therecipient's device; the referral matcher may perform analysis of contentthat is conveyed to the recipient to determine matching products,services, merchants, brands, or promotions for example associated withthe inferred reference in the communication.

In one or more embodiments, a referring user may be able to select text(or other items) in a communication, and initiate a search for listingsin the merchant database matching the selection.

One or more embodiments may use language processing and analysistechniques to identify words, phrases, text strings, or other elementsin the input that are associated with one or more products, services,merchants, brands, or promotions in the database. These techniques mayinclude for example, without limitation, natural language processing,collaborative filtering, artificial intelligence, affect analysis,type-ahead, predictive analytics, machine learning, recommendationengine, and personalization engine. One or more embodiments may processand analyze any other types of content, such as images, video, audio,speech, links, code, URLs, website addresses, links to mobileapplications, or animations, to identify items that are associated withone or more products, services, merchants, brands, or promotions in thedatabase. For example, image processing may identify images of itemsthat are associated with entries in the database or with categories ofentries. As an illustration, images in a message may be processed by aneural network or other classifier to identify the types of items in theimages; if a message contains images of a car, for instance, databaseentries corresponding to car sales or car rentals may be matched and oneor more associated referral links may be inserted into the message.

If multiple merchants in a database are identified by the referralmatcher as potential matches, based on the communication between theusers, the system may ask the referring user to select from among themultiple matches. In one or more embodiments, the system may select aspecific merchant automatically. Selection of a merchant may be based onany factor or factors on which merchants may be rated, scored, measured,or compared, including for example, without limitation, the size oramount of the referral credit associated with each merchant, thelocation of the merchant, the availability of inventory, speed offulfillment of orders, ratings or reviews related to the merchant, orthe price of the product or service offered by the merchant. Thereferring user and receiving user may both set preferences that forexample invoke a strategy pattern to determine the merchant to provide areferral to, or to generate an ordered list of merchants. For example,the system may automatically select the merchant having the largestreferral credit, or automatically select the merchant having the lowestprice, quickest delivery time, etc. In one or more embodiments, thereferral may be made based on the recipient's preferences, so that therecipient's favorite merchant's may be inserted into a list presented tothe referring user and/or by the receiving user in one or moreembodiments. In one or more embodiments, the referral may be made basedon business rules determined by the communication intermediary. Otherstrategy instances may be utilized to correlate the preferences of thereferring user with the preferences of the receiving user to find themost appropriate referral as well.

One or more embodiments may incorporate settings which enable thereferring user or the receiving user to control the frequency and mannerin which the referral matcher presents matching merchants and may promptthe referring user or receiving user to agree to the insertion of thereferral link.

In one or more embodiments, a referral link may initially direct arecipient to an intermediate server or system; this intermediate servermay then redirect the recipient to a target destination related to thereferral. The intermediate server may perform additional processing todetermine which target destination is appropriate or optimal for thereferral. For example, without limitation, in one or more embodimentsthe referral matcher executing on a referring user's device or coupledwith a network or messaging gateway may perform an initial match thatsimply identifies that a relevant product, service, brand, or merchantexists, or that selects a broad category or grouping of potentiallymatching products, services, merchants, brands, or promotions. Thereferral link associated with this initial match may direct therecipient to the intermediate server, which may then perform anadditional matching step to select a specific merchant, site, page, orother destination to complete the referral. The redirection linkgenerated to this final destination may contain the same or similartracking information (such as an identity of the referring user orcommunication intermediary) so that the referrer or communicationintermediary may obtain credit for a successful referral. Theredirection link may direct the recipient for example to any or all of aweb site, another server, another intermediate server, a web service, aspecific web page, an application, a URL, or a URI. A potential benefitof this two-stage matching process using an intermediate server is thatthe database accessible to the referral matcher can be smaller, and doesnot need to be updated in real time. The processing for the initialmatch may also be faster and less resource intensive and may reduce thelatency of the creation and transmission of the message. The matchingand final selection process on the intermediate server may access moredetailed information on products, service, or merchants, includingpotentially information that is updated in real time (such as merchantbids for referrals). This second stage of matching and selection mayalso utilize more computing resources available on a server or a networkof servers.

One or more embodiments may provide or access a dashboard or otherreporting system which enables the communication intermediary orreferring user to access information about the referral links sent, andmay include, without limitation, information regarding the amount ofreferral credit earned, the identity of the recipients which conductedtransactions, the identity of the merchants with which transactions haveoccurred as a result of their referrals, and the communicationsplatforms through which the referrals were sent.

One or more embodiments may present a referrer profile page which may bepublic facing, and which may display brand, product and servicereferrals with referral links. In one or more embodiments, this enablesa more graphical interface for receiving users to select referrals thatgenerate rewards for the referrer. In one or more embodiments, acommunication may be generated that includes a link to a site thatallows for graphical selection of a particular merchant for example bythe receiving user.

One or more embodiments may include a mechanism which enables thereferring user, while on a web page, to select the URL or otherindicator of such web page, and create a referral link to that pagewhich may include an identifier of the referring user, so that thereferral may be tracked, and enable the referring user to insert thisreferral link into a digital communication.

In one or more embodiments, a referral link may be, may contain or maylead to a coupon for a product or service, instead of or in addition toa link to a site. The coupon may be for example in the form of a code, aprintable document, a UPC code, a QR code, a promotional code, a ticket,an image, or another identifier. The recipient may use the coupon forexample for either online or offline transactions; in an offlinetransaction, the recipient may for example transact with a via aninteraction which does not get tracked via a link to a site.

In one or more embodiments, referral links may be added by acommunication intermediary in a communication path between a sender andreceiver. The intermediary may receive a communication from the senderto a receiver. The communication may contain any or all of an identifierof the sender, an identifier of the receiver, and a communication body.The intermediary may transmit the communication or any portion thereofto a computer that executes a referral matcher. For example, thereferral matcher may execute on a network gateway or on a server as aweb service. The referral matcher may be coupled to the communicationintermediary (and associated messaging gateway) associated with thesender, the recipient or both. The referral matcher may analyze thecommunication to determine whether there are any matches to listings inthe referral database. When a match is found, the referral matcher maytransform the communication to insert a referral link to a site or otherresource associated with the matched item from the database. The linkmay also contain an embedded referral tracking code that identifies thereferrer, which may be any combination of the sender, the receiver, orthe intermediary. The transformed communication may be returned to thecommunication intermediary, which transmits it along the communicationpath to the receiver. If a receiver uses the link and completes anassociated task (such as a purchase), a referral tracker may receive anotification of the transaction with the embedded referral trackingcode, and credit the referrer for the transaction.

In one or more embodiments, when a match is found, the referral matchermay convey to the communication intermediary instructions as to how totransform the communication to insert a referral link to a site or otherresource associated with the matched item from the database and embed areferral tracking code that identifies the referrer, which may be anycombination of the sender, the receiver, or the intermediary. Thecommunication intermediary may then execute the instructions andtransmit the transformed communication along the communication path tothe receiver.

A communication intermediary may be for example, without limitation, oneor more of a communications carrier, a wireless provider, a cellularprovider, a network provider, an internet service provider, a socialmedia platform, a messaging service, a telephone service provider, abroadband service provider, a Wi-Fi provider, an email service, a textmessage service, a mobile application, a software application, a chatservice, an application which conveys, transmits, receives, routes,forwards, stores, directs, edits, filters, or formats communications, ora gateway between any types of devices, networks, routers, or nodes. Acommunication may be or may contain for example, without limitation, oneor more of a message, a text message, an email message, a voice message,a video message, a website link, a link to a mobile application, apicture message, a transcribed message, a communication via socialmedia, a communication via a shopping site, a communication via amessage board, a posting to a product review service, an encryptedmessage, a digital communication, a comment posted to a digital mediaservice, and a communication via a messaging application. In one or moreembodiments, the receiver of a message may use a referral link byperforming one or more of a tap, click, gesture, response, userinterface interaction and verbal command. The associated task may be oneor more of a click, view, visit, transaction, purchase, reservation,subscription, sign-up, submission, software installation, download,inquiry, content consumption, survey completion, and participation in adigital interaction.

In one or more embodiments, the referral link may link for example,without limitation, to one or more of a website, a software application,an e-commerce service, a merchant shopping cart, a mobile application, acomputer application, a store, a redirector, a link-tracking service, anaffiliate network, a video player, a coupon or coupon code, a promotionor promotion code, a discount code, a transaction code, a mappingservice and a URL. The listings in the referral database may be forexample, without limitation, one or more of a product, a service, abrand, a merchant, a name of a merchant, a name of a web site, a name ofa product, a name of a service, a location, a review, a rating, aproduct number, a model number, a description, a picture, an image, adiagram, a barcode, a UPC number, an RF code, an activity, a keyword, aphrase, a product category, an SKU, an instruction, a suggestion, asolution, an information source, a person, an organization, and aprofessional. The database may be for example, without limitation, oneor more of a file, library, catalog, directory, open graph, real-timeweb search, cached web search result and data feed.

In one or more embodiments, the processing performed by the referralmatcher may determine whether content such as a word, phrase, textstring, URL, web site link, web site address, link to a mobileapplication, domain name, image, audio or video component, or code inthe communication body corresponds to a listing in the referraldatabase; if so, the link may replace or be inserted around thatmatching content. In one or more embodiments, processing may alsoperform sentiment analysis to determine the sentiment in thecommunication towards the matched item.

In one or more embodiments, the referral matcher may have a configurablecloseness-of-match parameter that determines how closely thecommunication must correspond to a database item in order to generate areferral link.

In one or more embodiments, processing may also determine whether acommunication indicates a subcategory associated with a matched item.Subcategories associated with database listings may include for example,without limitation, one or more of a size, a style, a model, a type, asub-brand, a feature, a name, a category, a quality, a stock keepingunit, a characteristic, a color, a date, a time, a location, and aquantity. If subcategory data is identified in a communication, thereferral link may include this data, for example by linking to aspecific web page or by incorporating URL parameters in the link.

In one or more embodiments, a referral credit may be provided to one ormore of the communication intermediary, the sender and the recipient.The sender's and recipient's identity may be encoded to protect privacybefore it is passed to the referral matcher.

In one or more embodiments, either or both of the sender or receiver ofa message may be able to send a message to opt-out of having referrallinks added to communications. For senders or receivers who opt out, thereferral matcher may not add links to messages for those users. One orboth of the sender or receiver may be able to send a message to modifycommunications preferences that affect when, how, or how frequentlyreferral links are added, how they are used, or how they are displayed.

In one or more embodiments, when a referral matcher identifies more thanone possible site that corresponds to a listing that is a match in acommunication body, it may select a site based on a performance metric,such as for example, without limitation, the amount of referral creditassociated with the site, a price associated with the site, a degree ofsimilarity between the site and the communication, a transactionconversion rate associated with the site, a defined set of businesslogic associated with the site, how closely characteristics of thesender or receiver correspond to the site, proximity of the site to thesender or receiver's location, a location associated with the site, aspeed of fulfillment associated with the site, a review score, apopularity score, or a rating score associated with the site.

In one or more embodiments, the referral link may link to anintermediate server, and the intermediate server may select a finaldestination when a message recipient clicks on or otherwise interactswith a referral link, and may then redirect the recipient to that finaldestination. To select a destination from multiple possibledestinations, destinations may be compared on a performance metric suchas the metrics described above.

In one or more embodiments, matching against a referral database may beperformed for any image contained in a message or otherwise shared ortransmitted from a sender to a receiver. Processing of an image todetermine whether a positive match exists may include detectingcomponents in the image and determining a type of each component,analyzing each component, and synthesizing component analyses to form adescriptor of the image. The descriptor may then be compared to thelistings in a referral database. In one or more embodiments, an optionalvalidation of a tentative match may be performed. (This validation stepmay not be performed in one or more embodiments or embodiments, or itmay be performed only in certain situations.) Validation may includeretrieving a second image associated with the matched item, andcomparing this second image to the original image to form a confidencescore. When and if validation is performed, a referral link may begenerated for the image if the confidence score exceeds a threshold. Anyof the steps for image analysis, matching, and referral link generationand presentation may be performed on the sender's device, the receiver'sdevice, a device used by the sender to encrypt all or a portion of acommunication, a device used by the receiver to decrypt all or a portionof a communication, any communication intermediary in the communicationpath between the sender and receiver, or any combination thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the inventionwill be more apparent from the following more particular descriptionthereof, presented in conjunction with the following drawings wherein:

FIG. 1 shows an example of a user using an embodiment of a messaginggateway monetization system that accepts input from a referring user andautomatically determines or otherwise infers merchants associated withproducts and/or services to refer based on the input, i.e., to matchreferrals with the input, and insert the referral into a message to arecipient that contains a trackable product or service referral.

FIG. 1A shows a variation of the example of FIG. 1, where the referralmatching system operates in a manual mode that requires user actionbefore showing suggested referrals.

FIG. 2 continues the example of FIG. 1, showing the recipient receivingand using the referral, and the system crediting the referrer for thesuccessful referral.

FIG. 3 shows illustrative options of the types of credits andnotifications the referral system may provide to the referrer.

FIG. 4 illustrates an embodiment that allows a user to select text in asocial media posting, and search for referrals matching the selectedtext.

FIG. 5 illustrates an embodiment that generates a referral using a sharebutton.

FIG. 6 illustrates an embodiment that matches an image captured by acamera (such as a barcode or an image of a product) and that generates areferral based on this image.

FIG. 7 illustrates an embodiment that uses natural language processingto identify words in a communication that may infer or otherwise matchone or more merchants, and that presents a choice of matching merchantsto the user.

FIG. 8 shows an illustrative process flow for an embodiment of apeer-to-peer trackable referral system, where the input used for thereferral lookup is obtained from a keyboard.

FIG. 8A shows a variation of the process flow of FIG. 8, where areferral link directs a recipient to an intermediate server that makes afinal selection of a referral destination, and redirects the recipientto that destination.

FIG. 9 shows illustrative alternative embodiments for obtaining inputthat may be used to generate a referral.

FIG. 10 shows illustrative alternative embodiments for identifyingpotential referrals.

FIG. 11 shows illustrative alternative embodiments for informationsources that may be used for a database of merchants for potentialreferrals.

FIG. 12 shows a variation of the process flow of FIG. 8, where input fora referral is received from an image instead of from a keyboard.

FIG. 13 shows a variation of the process flow of FIG. 8, where areferral is initiated via a share button.

FIG. 14 shows a variation of the process flow of FIG. 8, where areferral is initiated when a user selects text or a URL and searches formatching potential referrals.

FIG. 15 shows an illustrative system with a referral matcher thatoperates at a gateway, rather than on the sender's device, and thatidentifies a communication intermediary as the referrer.

FIG. 15A shows a variation of the embodiment of FIG. 15, where both thesender and the intermediary are identified as referrers.

FIG. 15B shows another variation of the embodiment of FIG. 15, where thegateway that invokes the referral matcher is associated with thereceiver's communications carrier rather than with the sender'scommunications carrier.

FIG. 15C shows another variation of the embodiment of FIG. 15, where asender copies a URL into a message, and the referral matcher identifiesan entry in the referral database that matches this URL.

FIG. 15D shows another variation of the embodiment of FIG. 15, where thereferral matcher generates transformation instructions that are executedby the communications intermediary to modify a message to insert areferral link.

FIG. 16 continues the example of FIG. 15 to illustrate crediting thereferring communication intermediary when the recipient clicks the linkand makes a purchase.

FIG. 17 shows an illustrative embodiment that performs sentimentanalysis on a message to avoid adding a link to a target that themessage refers to negatively.

FIG. 17A shows an illustrative embodiment that analyzes the context of amatching item, such as nearby words or phrases, to determine whether toa referral link.

FIG. 18 shows an illustrative embodiment that provides a configurabledegree of closeness of match for the referral matcher.

FIG. 18A shows an illustrative embodiment with multiple sites that matcha database listing; the referral matcher selects a specific site fromthe multiple options based on performance metrics such as thereimbursement for a successful referral.

FIG. 19 illustrates an embodiment that searches a message forsubcategory information related to a primary target in order to generatea more specific referral link.

FIG. 20 shows an illustrative embodiment that lets a sender opt-out ofhaving referral links added by a communication intermediary.

FIG. 21 shows an illustrative flowchart of steps that may be used toanalyze images and generate referral links based on this analysis.

FIG. 22 shows analysis of an illustrative image and generation of areferral link for this image.

FIG. 23 shows a variation of the example of FIG. 22 where multiple itemsfrom the referral database match the image, and a prioritization andselection step is performed to determine which match or matches to usefor referral links.

DETAILED DESCRIPTION OF THE INVENTION

A monetization system for images will now be described. In the followingexemplary description, numerous specific details are set forth in orderto provide a more thorough understanding of embodiments of theinvention. It will be apparent, however, to an artisan of ordinary skillthat the present invention may be practiced without incorporating allaspects of the specific details described herein. In other instances,specific features, quantities, or measurements well known to those ofordinary skill in the art have not been described in detail so as not toobscure the invention. Readers should note that although examples of theinvention are set forth herein, the claims, and the full scope of anyequivalents, are what define the metes and bounds of the invention.

FIG. 1 shows an illustrative embodiment of the invention, illustratingthe generation of a referral for example as part of a peer-to-peercommunication. (FIG. 2, described below, continues this example to showhow a referral is processed by the system when used by a recipient.)User 101, the referring user, uses mobile device 102 to generate acommunication to one or more other users. In this example, user 101composes a text message to user 121. One or more embodiments mayintegrate referrals into any type of communication and independent ofthe application used, including but not limited to text messages,emails, SMSs, voice messages, picture messages, video messages, chatmessages, Twitter® messages, Instagram® messages, messages sent viamessaging platforms including for example without limitation WhatsApp®,Snapchat® or Facebook® Messenger, and postings on any websites orservices including for example, without limitation, social media sitessuch as Facebook®, Tumblr®, Google+®, YouTube®, Vine®, Pinterest®,Flickr®, or LinkedIn®. In one or more embodiments users may use any typeor types of devices to send or receive communications and to generate,receive, or use referrals, including but not limited to a mobile devicesuch as device 102. For example, without limitation, users may usemobile devices, cell phones, smartphones, tablets, laptop computers,notebook computers, desktop computers, server computers, smart speakers,smart watches, wearable devices, communication devices built intovehicles, smart glasses, augmented reality devices, content managementsystems, virtual reality devices, or networks of any of these devices.Users generating referrals may use the same devices or different devicesfrom those used by users receiving and using referrals.

A referring user and a receiving user may each be any person, system,organization, group of persons, robot, application, business, or agencythat communicate in any manner over any medium. For example, withoutlimitation, a referring user may be an individual, a professionalcontent creator, an editor, an author, or a business. A user may beacting on his or her own behalf, or on behalf of a business, anorganization, or an agency. Recipients of a communication from areferring user may be peers, family, friends, known or unknown personsor groups, readers, business contacts, or followers of a feed or blog.Recipients may be individuals or they may be groups or audiences of anysize and composition. Although the term receiving user is used in thesingular in certain scenarios herein, receiving user means one or moreusers that receive a communication from the referring user.

In the example shown in FIG. 1, user 101 is typing a message with thecursor 105 midway through a word. The user types the message using avirtual keyboard 104 as utilized in one or more embodiments on themobile device or electronic device 102. In this embodiment, the system,which includes virtual keyboard 104 accepts the input, here textcharacters and transmits the text characters to a referral matcher 110.Referral matcher 110 receives input from the virtual keyboard 104, andmay suggest potential referrals based on the keystrokes received fromthe keyboard. The referral matcher may execute directly on the mobiledevice 102, or it may execute on another device (such as a server or acloud or Internet accessible service) remote from the mobile device 102,or any combination thereof. The referral matcher 110 accesses a database111. The database may contain for example, without limitation, products,services, merchants, brands, promotions, locations, places of business,stores, events, persons, performers, or any related characteristics. Inone or more embodiments, a computer or server that executes the specificinstructions to implement the referral matcher may also host thedatabase or in other embodiments, the database may be remote to thecomputer that hosts the database. In one or more embodiments, thereferral matcher may access any number of data sources to identifypotential referrals. These data sources may be local to the mobiledevice (or other device) 102, or remote from the mobile device (forexample, on a server accessible via the Internet or via any othernetwork connection). Database 111 may contain any information that maybe used to locate or create a referral, or to select from or prioritizeamong multiple matching merchants, such as for example, withoutlimitation, the name or names of a merchant, the name or names of a website, the name or names of products or services, brand names, thelocation of merchants, reviews or ratings of products, services, ormerchants, product numbers, SKU numbers, model numbers, descriptions,pictures, images, diagrams, barcodes, UPC numbers, RF codes, the amounta merchant pays for referrals, information on any promotions, or anydata derived from or related to any of this information, including anygeneric categories that may infer one or more merchants, for exampleairline or flight or fly to infer a particular merchant that providesair travel services.

Database 111 may for example include information about andcharacteristics of merchants, such as merchant names, ratings, reviews,conversion rates, locations, and hours. A merchant may be for example,without limitation, a company, organization, individual, or other entitythat makes, sells, or distributes products or services available for auser to purchase, consume, or enjoy. A merchant may be for example,without limitation, a retailer, a service provider, an e-commerceservice, a sales agent, a salesperson, a manufacturer, or a distributor.A merchant may be associated with an intermediary. Database 111 may forexample include information about one or more intermediaries. Anintermediary may be for example, without limitation, an agency, amarketing firm, an affiliate network, a referral marketing provider, orany other individual or entity that represents a merchant and to which alink may be directed or through which a link may be redirected. Database111 may for example include information about products and services,such as the products and services provided by a merchant or anintermediary. This information may include for example, withoutlimitation, product names, descriptions, alternate references (forexample “flights to Denver”), pricing, and inventory availability.Products and services may be for example, without limitation, any goodthat a user can purchase, enjoy, or consume; these may be physicalproducts, virtual products (such as digital content, movies, music, orother formats), or services. Database 111 may for example includeinformation on one or more brands, such as for example, withoutlimitation, a trade name of a product, service, manufacturer, marketer,or retailer. Database 111 may for example include information on one ormore promotions, such as for example, without limitation, informationregarding discounts, special sales, cash-back offers, and new userrewards, which a merchant may offer relating to sales of its productsand services.

In the example of FIG. 1, referral matcher 110 analyzes keystrokes fromvirtual keyboard 104 and compares these keystrokes to the names andother information in database 111, and determines that item 107 is apotential match. In a simple example, “Acm” may be compared to any itemsin database 111 that being with “Acm” such as “Acme”, “Acme Airlines” orthat is associated with an acronym for text in the database, i.e., “ACM”that standards for “Association of Computing Machinery”. In otherembodiments, words such as “flight” may be associated in database 111with all merchants that are airlines. Any schema may be utilized toprovide potential matches and contain any number of associations foreach merchant, product or service to compare with. It therefore providesthis item as a suggested completion for the virtual keyboard input. Thevirtual keyboard application may also provide other suggestedcompletions such as completion 106, which may not correspond to areferral. Suggested referrals may be identified for example using anicon such as icon 108. If the user 101 selects the suggested referral107, which may be performed in one or more embodiments with a single tapon the suggested referral, a referral link 124 with this referral may begenerated and placed into the message 123. In other embodiments, thelink may be automatically inserted based on preferences or any strategypattern employed by the system. After the user completes and sends themessage, user 121 receives the message on device 122. The message 123includes the referral link 124 generated by the referral matcher 110.

In the example shown in FIG. 1, the virtual keyboard application linkedto the referral matcher 110 is used as input for a text messagingprogram. This same virtual keyboard application may be used universallywith any other applications or services on mobile device 102, providingthe same referral matching capabilities for these other applications orservices without requiring altering each and every differentcommunication application to including the trackable referral generationcapability of embodiments of the invention. This flexibility illustratesa benefit of the system in providing a potentially general-purposereferral capability that may be used across multiple applications andservices. For example, without limitation, the same keyboard 104 withreferral icon 108 and referral matcher 110 may be used for an emailapplication, a web browser, a chat application, a blogging application,a publishing application, a word processing application, or any otherapplication used by user 101. In any of these applications, the referralmatcher may analyze input from the keyboard and suggest potentialreferrals to be embedded in communications. This universal ability togenerate trackable referrals regardless of the application in use in apeer-to-peer setting is unknown in the art and makes referral generationand tracking pain free and extremely easy to use for referring users.

In one or more embodiments, the referral matcher may be incorporatedinto an “app extension,” including but not limited to a keyboardextension. This app extension may for example provide enhanced oralternate capabilities for user input features. For example, for akeyboard extension, the system may provide enhancements or replacementsfor keyboard features such as auto-correction, spell-checking,auto-completion (of words or phrases, using for example predictive textanalysis), and databases of words and phrases such as a lexicon,dictionary, or thesaurus. In one or more embodiments, the referralmatcher may be integrated within or coupled to an “app extension,”including but not limited to a keyboard extension, such as for exampleSwiftkey®, iOS®, Swype® or Gboard®; the referral matcher may performanalysis of content that is input to such an app extension to determinematching products, services, or merchants, brands, or promotions forexample associated with the inferred reference in the communication. Forexample, for a keyboard extension, the referral matcher may provide analternate or complementary dictionary or lexicon for auto-completion,auto-correction, predictive text or spell-checking. In this example, thereferral matcher may analyze input from a referring user, it may thenanalyze this input to identify one or more products, services,merchants, brands, or promotions in the database that match the input.The referral matcher may then automatically generate a referral link andmay insert this referral link into the communication. The referralmatcher may automatically insert the referral link into thecommunication, or the referral matcher may prompt the referring user toagree to the insertion of the referral link and may then insert thereferral link into the communication.

The scenario illustrated in FIG. 1 is an example of the system workingin “auto mode,” where referral matches are automatically presented tothe user as they are discovered by the referral matcher. One or moreembodiments may also support a “manual mode,” where the matcher 110 forexample may work in the background but may not automatically showsuggestions to the user 101. Instead, in manual mode, the keyboard oranother element of the system may provide an indication or cue toindicate to the user 101 that there is a potential match. FIG. 1Aillustrates an example of an embodiment operating in manual mode. User101 configures the system to switch from automatic mode 1A01 to manualmode 1A01 a. (This configuration may be performed for example using asettings screen or any other method of modifying the configuration ofthe referral system or the keyboard.) As the user uses virtual keyboard104 to type input 1A02 as part of a message, the referral matcher mayoperate in the background to locate potential referrals related to themessage, but it may not display these referral matches unless and untilthe user requests to see them. Text completion suggestions such as 1A04may not contain referral suggestions initially; instead they may besimple text completions for the current word 1A03, rather than referrallookups. However, the referral matcher may provide an indication that ithas located one or more matching referrals, in this example by changingthe appearance of icon 108 a (compared to the original icon 108 in FIG.1). For example, without limitation, icon 108 a may flash, blink, changecolor, change size, change shape, or otherwise change any aspect of itsappearance as a notification of potential referrals. This changed iconsignals to the user that potential referrals are available for review.In manual mode, the user 101 may for example single tap the icon 108 ato indicate that the user wants to see the potential referral orreferrals. In this illustrative example, the referral matcher locatestwo potential referrals, one for an airline and one for a hotel, byscanning and analyzing the input 1A02. A first tap on icon 108 a showsthe first potential referral 1A10 in the word completion area of thevirtual keyboard. The user may for example tap on this referral 1A10 toinsert it into the message 1A02. In this case, the appearance of icon108 b shows that there are more potential referrals. If the user tapsicon 108 b again, a second referral 1A11 is displayed. Again, the usermay tap this suggestion 1A11 to insert the referral into the message1A03. Icon 108 c returns to its “normal” state (as in FIG. 1),indicating that there are no more potential referrals that have beenlocated in this message text. This example is illustrative; one or moreembodiments may use any type of user interface to present referral matchsuggestions to a user either automatically or in response to useractions. For example, a user interface may show all possible referralmatches to a user simultaneously, instead of cycling through them asillustrated in FIG. 1A. One or more embodiments may also provide optionsfor a user to select among several referrals, which may for instancecorrespond to different product categories. For example, the word“china” in a message may refer to the product category porcelain, or thecountry (which may suggest referrals related to travel); in one or moreembodiments, the system may prompt the user to select between theseoptions (although in some cases, embodiments of the system automaticallydetermine the appropriate product category by analyzing the messagecontext or historical data regarding the user's inputs).

One or more embodiments of the invention may analyze the user's inputand manipulate the input to transform or augment references to itemsinto any type of reference or trackable link. For example, the systemmay replace or augment an explicit item reference or an implicit itemreference with one or more of: a hyperlinked version of the samecontent, a URL immediately after the matching content, a footnote-stylereference to the item, or with any other format that contains atrackable link or similar reference. In each case, the system mayreplace or augment the contents of the message while keeping theoriginal meaning of the content intact. In one or more embodiments,replacing or augmenting of message contents may be done automatically,either as the message is generated or prior to transmitting the message.In one or more embodiments, the system may offer to optimize the user'scommunication before it is sent, potentially at the user's explicitrequest. This optimizing of the user's communication may for exampleaugment the communication such that a reference to an item includescharacteristics that include tracking and attribution (such as the userID and the item in the database).

FIG. 2 continues the example of FIG. 1. User 121 first receives message123 containing referral link 124, as described above. The referral linkmay for example contain a link to a web site or site or an e-commercesite corresponding to the referred merchant, product, or service. One ormore embodiments may use any type or types of links or references,including for example, without limitation, a hyperlink, a URL, ashortened URL, a full URL, an image, a sound, a video, a short code, ora hashtag. In the example shown in FIG. 2 the link is a hyperlink forillustration. In one or more embodiments, this hyperlink may link to anintermediate web site or server, for example, which may be used fortracking or accounting, and which may then forward the user to theappropriate e-commerce site. In the example of FIG. 2, receiving user121 selects referral link 124, which takes the user to the website 202(or to a similar service or application). User 121 may activate thereferral link 124 via any method, including for example, withoutlimitation, a click, a tap, a verbal command, a gesture, or any otheraction or prompt. The referral link 124 includes URL 201 that has theaddress of the e-commerce site (or to another server that then forwardsto this e-commerce site), along with a parameter that identifies thereferring user. This URL format is illustrative; one or more embodimentsmay embed any desired information into a referral link in any desiredformat. For example, without limitation, URL 201 may encode or includeany combination of one or more of an identifier that identifies thesending user, the characteristics of the content that the user input,the matching item from the database, a merchant to which the recipientis directed, the context within which the content was input, and theapplication that the user used to input the content.

URL 201 may be formatted in any desired manner, using any desiredencoding scheme or schemes to embed the desired information. Forexample, the system may include within the database a set of URLformatting rules or templates which may identify the technicalrequirements for URLs or encoded information for each merchant, product,service, brand, promotion, intermediary or type of reference made by theuser. When the system creates a link, the system may perform a lookupwithin the URL formatting rules to determine and execute the appropriateURL structure to successfully send the recipient to the correct site orcoupon with functioning tracking and attribution. The URL may beconstructed to send the recipient to a deep-link within the merchantsite which corresponds with the reference made by the user and which mayinclude, for example and without limitation, a product detail page,search result or a category page. The URL may be constructed to map theuser's input to an item and merchant URL prefix so that the merchant canreceive the intent of the user. For example, the user may input “strappysandals at BigMegaStore” and the system would generate a URL with a linkto the merchant site (such as www.bigmegastore.com) and a reference to“strappy sandals” within the URL, for example via a search term.

User 121 may then interact with site 202 to purchase goods or servicesor perform other transactions. When the user completes a transaction,for example using button 203, referral tracker 210 receives thisinformation and credits the referring user with the successful referral.In one or more embodiments, information relating to completedtransactions may be obtained from one or more third parties, includingfor example the merchant, an affiliate network, a credit card processor,or other system. For example, the website 202 may transmit a message toa referral tracker server with the URL 201 (so that the originalreferrer can be identified); this message may also include anyadditional details of the transaction. The referral tracker 210 maydetermine the amount and type of referral credit 211, based for exampleon the amount of the transaction and on specific arrangements with thee-commerce merchant for referrals. The referral tracker may then collectthis credit from the referring merchant 212, and transmit this credit tothe original referring user 101 (possibly net of a fee to the referralsystem provider). In one or more embodiments, the computer or serverthat executes referral matcher 110 may also execute or otherwise hostreferral tracker 210. In other embodiments, a distributed architecturemay be utilized and multiple computers may implement the referralmatcher 110 and referral tracker 210. Any cookie based technique or anyother technique may be utilized to provide referral tracking so that forexample a receiving user may use a link or otherwise purchase a productor service at a later date and still be tracked as taking an actionbecause of the referral, so that the referral tracker may credit thereferring user.

In one or more embodiments, any or all of the functions of the referraltracker 210 may be provided by third-party systems or services. Forexample, without limitation, these third-party systems or services mayprovide some or all aspects of tracking, attribution, paymentprocessing, calculation of referral credits earned, or reconciliation.

In the example illustrated in FIG. 2, link 124 leads directly to amerchant site at URL 201. A site referred to via a link may be forexample, without limitation, a website, a mobile application, a desktopapplication, a server, an automated communications platform, a callcenter, or a salesperson (for example via telephone, online chat, textcommunication, or any other form of communication). In one or moreembodiments, a link may route through an intermediate server, which mayperform functions such as for example, without limitation: determiningwhich merchant or merchants provide the item; prioritizing whichmerchant site the system should direct the recipient to; identifying apromotion or coupon for a product, service, brand, or merchant; andprioritizing which merchant coupon to make available to the recipient.In one or more embodiments, when a recipient activates a link the systemmay generate and present to the recipient a coupon. The coupon may befor example in the form of a code, a printable document, a UPC code, aQR code, a ticket, an image, or another identifier. The recipient mayuse the coupon for example for offline transactions, in which therecipient transacts with a merchant (physically, verbally, or digitally)via an interaction which does not get tracked via a link to a site. Thesystem may present the coupon in any desired format, including forexample in a printable document, a virtual document, or in an audio orvideo format.

One or more embodiments may provide a referral credit for any type ofaction or event that results from a referral, including but not limitedto a purchase transaction as shown in FIG. 2. For example, withoutlimitation, a referral credit may be provided for a completedtransaction by a recipient, for a recipient clicking on a link, for arecipient consuming any product or service, or for a recipient taking anaction after clicking on a link, such as signing up for or joining aprogram or service, requesting information, enrolling, applying,registering, subscribing, or installing or downloading an application.Any type of transaction may be tracked and credited, including offlinetransactions that may for example include a purchase at a physicalstore, consumption of products or services offline (for example at arestaurant, a physical therapy site, or a live entertainment site).Transactions may include for example, without limitation, purchases,clicks, downloads, submittals, installs, sales inquiries, views,rentals, one-time purchases, recurring purchases, and subscriptions.

When a user completes a creditable transaction, a merchant associatedwith the transaction may owe a fee. The fee may be in any form includingfor example, without limitation, money, credit, points, discounts,cash-back, or membership status. The merchant (or their agent orintermediary) may remit the fee or fees to the system. The system maythen remit all or a portion of the remitted fees to the user, to therecipient, or to another designee that the user or recipient maydesignate (such as a charity, organization, or individual). Remittingmay occur via any mode, depending on the nature of the fee, and mayinclude for example, without limitation, ACH, EFT, wire transfer, acheck, a digital wallet, Bitcoin or other digital currency, crediting acredit or debit card or gift card, remittance via online paymentproviders such as PayPal® and Venmo®, crediting points to a loyaltyprogram, and providing a discount code for future purchases.

In one or more embodiments, a referral credit may be provided to anyperson, persons, groups, or organizations, including but not limited tothe original referrer. A referral credit may be monetary, or it may takeany other form such as an award, a gift of goods or services, a creditagainst previous expenditures, or a credit for future expenditures orusage. FIG. 3 illustrates several options for referral credits; theseoptions are illustrative and are not limiting. Referral tracker 210 mayissue a monetary credit 302 to the original referrer. It may generate afeedback message 301 to the referrer, which may for example inform thereferrer when a referral has been used by a recipient; feedback messagesmay also provide the referrer with aggregate information on all activityresulting from the referrer's referrals. Feedback messages may includeinformation such as the time of a recipient's transaction, therecipient's transaction amount, the referral fee amount earned, the rateof links sent to conversions (i.e., a quality score for a user), and asettlement status change. One or more embodiments may also provide anaccount profile or account summary view accessible to the referringuser, such as a user dashboard, where the user may view information suchas details of referrals, resulting clicks, transaction, credits,payments, products, services, or merchants referred, conversion ratesfor referrals, analysis and suggestions which aid in optimizing futurereferrals, and preference settings. A user dashboard may also providemechanisms to control the modes in which fees are remitted or stored,such as for example by physical check, ACH, international wire, localbank transfer, cash pickup, PayPal®, Venmo®, pre-paid debit card, orgift card. Referral tracker 210 may generate a referral credit 303 inthe form of a donation to an organization, for example which may or maynot be made in the name of the referrer or may otherwise be associatedwith the referrer. In one or more embodiments referrals may be trackedon a group basis; for example, referral tracker 210 may generateaggregate referral credits 304 to a group (or on behalf of a group)based on activity resulting from referrals by any or all members of thegroup. In one or more embodiments, a referral credit may also be sharedbetween a referring user and the receiving user (such as referring user101 and receiving user 121 in FIG. 1.)

In one or more embodiments, the system may enable a user to create (ormay automatically create) an interface (a “user page”) that aggregatescontent and links that a user has input into an application via thesystem. The user may be able to curate the links on the user page andmay be able to make the user page available or point recipients to theiruser page (for example via sharing of a URL or via any other digitalcommunication). If a recipient activates a link on a user page, the linkmay lead to a merchant site or generate a coupon with which therecipient may transact and generate a fee for the user (as describedabove).

In one or more embodiments, the system may provide a merchant dashboardthat may for example include settings, bidding controls, reporting, andother features that enable a merchant to manage its interactions withthe system and with users of the system, and to manage promotionalprograms.

In one or more embodiments, the system may enable a referrer to refer anew user to use the system or to use an application into which thesystem is integrated. The referrer may be an individual, company,organization, application, or other entity. The tracking code or otheraspect of the system may encode a user ID to enable tracking of linksgenerated by user content, so that when a user who uses the system asthe result of a referral from a referrer earns a fee, the referrer mayearn a portion of such fees, and the system may automatically remit thereferral fee to the referrer. Referrers who refer new users to thesystem may therefore earn a portion of the resulting fees generated bythese new users.

FIG. 4 illustrates an embodiment that allows a user to select text orother items and to initiate a search for potentially matching products,services, or merchants for referrals. In this example, the user iscommunicating via a posting 402 on a social media site 401. Any otheruser who reads the posting may receive this communication, and maythereby receive and use referrals embedded in the posting. The usertypes the posting and selects text 403, and then initiates a search formatching referrals by pressing icon 404, which may for example beintegrated into a virtual keyboard or otherwise made available to theapplication used for creating or reviewing the posting. The referralmatcher 110 matches the selected text 403 to the merchant database 111,and generates a referral link 201 that is inserted into the posting 402.The display name of the referral link may remain the same, such asremaining the selected text 403; one or more embodiments may modify thedisplay name of the referral link in any desired manner, for example asnecessary to comply with policies or technical requirements of thecommunications channel being used. In one or more embodiments, a usermay be able to select and match any information, including for example,without limitation, text, documents, files, web pages, URLs, images, ormessage threads, and may be able to initiate a search by the referralmatcher against any of this information.

In one or more embodiments, the referral matcher may perform a searchfor matches continuously in the background, and it may display theresults of these background searches when the user explicitly indicatesthat he or she wants to see the matches. For example, the user may tapan icon such as icon 404 in FIG. 4 in order to see matching referrals.Referral matches may be determined based on user selected text (as shownin FIG. 4), or based on automatically detected relevant content (asshown in FIG. 1), or on any combination thereof. In an “auto mode,”matching referrals may be shown to the user automatically as they arelocated, as shown in FIG. 1. In a “manual mode,” matching referrals maybe shown on request. In one or more embodiments, a user may be able toview multiple matching referrals on request, for example by successivelytapping on an icon to see the next match in the list. As anillustration, for message 402 in FIG. 4, if the system is in manualmode, a first tap on icon 404 may present a suggestion of Acme Airlines,while a second tap on icon 404 may present a suggestion of a hotel inTahoe.

FIG. 5 illustrates an embodiment that allows a user to generate areferral link using a share button or a similar sharing capability. User101 is browsing website 501, for example while making or planning apurchase. The user decides to share the website, and possibly also sharespecifics of the user's browsing (such as potential purchases), bypressing the share button 502. This share button may for example presenta sharing menu 503 to the user, which offers multiple ways to share thecurrent context. In one or more embodiments, the system may provide areferral sharing option 504 as an option in this sharing menu. When theuser presses the referral button 504, the referral matcher 110 mayexamine the current context that is being shared, and may determinewhether any items from database 111 match this context. If a match isfound, it may for example be inserted as a referral link 505 into amessage, which the user may then complete and send to any desiredrecipient or recipients. In one or more embodiments, matching of thecontext to potential referrals and generation of referral links may becombined with sharing via any other method, such as via a text, email,messaging app, or a social media site.

In one or more embodiments, sharing of a website URL or otherinformation may be done first via a sharing button, and then the optionto convert this link to a referral link may be presented in themessaging application that shares the link. For example, if user 101selects the email icon from sharing menu 503, an email application mayappear allowing the user to compose a message with the link to site 501.In this email application, the user may have an option to convert thiswebsite link to a referral link, for example by pressing a referral iconsuch as icon 504 that may appear in the email application, or, in theevent that the application launches a virtual keyboard, by pressing akeyboard icon.

In one or more embodiments, sharing of a referral related to a websitemay be performed directly using a button or icon that generates areferral link to that website, or to a similar site related to thewebsite. For example, without limitation, any website may incorporate a“refer” button or icon. When a user presses or otherwise accesses thisbutton or icon, a referral link may be created for a merchant related tothe website, where the referral link also identifies the user making thereferral. This referral button or icon may be analogous for example to aPinterest™ pin button, but instead of “pinning” a link to the website,the referral button or icon generates a referral link to the web sitethat will generate a referral credit to the referring user when and ifanother user accesses the referral link and completes a transaction.

FIG. 6 illustrates an embodiment that supports input of an image into areferral matching process. The system may match any image or any dataderived from an image with the items in a merchant database to obtainpotential referrals that match the image. The image may be for example,without limitation, a barcode (either a 1D barcode or a 2D barcode suchas a QR code), an image containing a product name or product identifier,a picture of a product, or one or more frames from a video containing aproduct. In the example shown in FIG. 6, user 101 uses mobile device 102to compose a message to a recipient. The user wants to insert a referrallink to a product, and therefore presses image capture button 601 toinitiate image capture 602, for example via a camera integrated intodevice 102 or accessible via device 102. This image may be for example,without limitation, a QR code 603 that identifies the product, service,or merchant being referred, or a picture 604 of the product, service, ormerchant. The image capture 602 provides the captured image or images tothe referral matcher 110, which compares the captured image or images toinformation in the merchant database 111. Comparison and matching ofimages may in one or more embodiments use external databases orservices, such as for example a lookup service for barcodes or anexternal database of product images. If a match is located, the referralmatcher may generate a referral link 124 and insert this link into themessage 123 transmitted to the message recipient 121. In one or moreembodiments, a barcode or QR code application may execute on electronicdevice 102 and the information obtained from the barcode or QR code maybe utilized to index into database 111 to determine a correspondingmerchant. In other embodiments, any image processing based applicationthat can recognize objects may be utilized to determine a product orproduct category that is utilized to search database 111 for.

In one or more embodiments of the invention, any type of input may beprovided to the referral matcher, including but not limited to textinput and image input. For example, without limitation, input into thereferral matcher may include sounds of any type or voice commands, andthe referral matcher may for example use voice recognition or any typeof audio processing to recognize the input and compare it to items inthe merchant database. Input may also include data captured by any typeof sensor, scanner, or reader. For example, one or more embodiments mayallow a user to use an RFID reader to read an RFID tag that identifies apotential referral.

In one or more embodiments, the referral matcher may use languageprocessing and analysis techniques to understand the user's input and todetermine matching merchants. These techniques may include for example,without limitation, artificial intelligence, natural languageprocessing, collaborative filtering, type-ahead, predictive analytics,machine learning, recommendation engine, personalization engine, or anycombinations thereof. FIG. 7 illustrates an embodiment where thereferral matcher 110 a has a natural language processor (NLP) 705integrated into or accessible to the referral matcher. The user's input701 is provided to the natural language processor 705, eitherautomatically as the user types or in response to the user pressing areferral button such as 704. In this illustrative example, the NLPsubsystem 705 recognizes the phrase 702 as relating to airline travel,and therefore searches the merchant database 111 a for airlines. FIG. 7also illustrates that in one or more embodiments the merchant databasemay be categorized, indexed, or organized in any desired manner tofacilitate referral matching; for example, database 111 a has airlinemerchants 706 identified as a particular product group. In this example,the natural language processor also detects that phrase 703 in theuser's input 701 indicates the user's likely destination for air travel.Therefore, the NLP subsystem 705 determines that the user is discussingcontext 707, which includes both the product category (airlines) and thelikely destination. All contextual information from analysis of theuser's input may be used to select one or more matching referrals.Virtual keyboard 104 may be utilized with the microphone instead ofdirectly manually typing as per FIG. 1 and also FIG. 7 for example.

Continuing the example shown in FIG. 7, in this case the referralmatcher 110 a determines that there are multiple possible merchants thatmatch the user's input. When multiple merchants match the input, thesystem may either automatically make a selection of a merchant to refer,or it may ask the user to make a selection. (One or more embodiments mayuse combinations of these methods.) FIG. 7 illustrates an embodimentthat asks the user to select which merchant to refer. Once the referralmatcher has determined matching merchants, the system presents screen710 to the user and asks the user to select one of the matchingmerchants. (This screen 710 is illustrative; one or more embodiments maypresent matching options to a user in any desired format and sequence.)In one or more embodiments, the referral matcher may provide additionalinformation to assist in making the selection. For example, in FIG. 7the selection screen 710 contains table 711 that shows the matchingmerchants along with the referral credits offered by each merchant.Table 711 also shows the price from each merchant for travel to thedestination 707 determined by NLP subsystem 705. This exampleillustrates how the complete context 707 determined by the NLP subsystemmay be used to determine an appropriate or potential referral, or toassist the user in making a selection from multiple matches. The usermay select a particular merchant from table 711 and generate a referrallink 201 a using button 713. Other user interface options may includefor example tapping a selected merchant in the table to generate areferral to that merchant, or tapping a button or icon to prompt thematcher to prompt the matcher to display a list of alternate matchingmerchants, or in some cases alternative product categories when theinput has matches in multiple categories (such as a book title that hasalso been made into a movie and a board game).

Although FIG. 7 illustrates an example where the user makes a selectionof a referral, in one or more embodiments the system may automaticallyselect a referral based for example on the referral credit, the price ofthe product or service offered, or on any other factors. One or moreembodiments may include a bidding system whereby merchants bid forreferrals, for example by offering referral credit rates or otherrewards. In one or more embodiments, the bidding system may be dynamic,in that referral credit information may be obtained for eachtransaction, for example whenever a referral matcher or a user is makinga selection among merchants to generate a referral.

One or more embodiments may include a merchant bidding system throughwhich participating merchants may compete with other participatingmerchants to be positioned higher in the prioritization of referrals. Amerchant may include, without limitation, a retailer, e-commerceprovider, service provider, advertiser, aggregator, broker, agency,promoter, or a party acting on their own behalf or on behalf of anotherparty. Any of the following techniques may be utilized. Embodiments ofthe system may provide merchants with a self-service system for placingbids. Merchants may bid on a variety of matching characteristicsincluding, without limitation, brand, product, service, keyword, phrase,product category, SKU or other product coding. Merchants may bid basedon bidding strategies including, without limitation, referral feepercentage, fixed amount of reward, bounty for leads, price per click,bounty for installation of application or software, or another actionperformed by a recipient who has utilized a referral link to themerchant. Embodiments of the system may include a bidding platform thatmay provide the merchant with multiple modes of bidding including,without limitation, manual mode (with which, for example, merchant setsa specific bid price for a specific product referral) or automatic mode(with which, for example, merchant designates a daily budget and timeperiod and the system adjusts the referral fee bid automatically todeliver the most referrals possible within the merchant's designatedbudget and time period). The merchant may specify bid pricing orreferral limits based upon specific characteristics of the sender orrecipient(s) including, without limitation, the sender or recipient'sgeography or, demographics, sender or recipient transaction history,input type used by sender, device type used by sender or recipient,communication channel through which the referral link was sent, or anycombination thereof. The merchant may set limits on bids placed and mayset time period for which the limits apply. Limits may include (withoutlimitation) maximum amount of referral credits paid (for example a dailybudget), maximum number of referrals received or other limits.Embodiments of the system may enable the merchant to set dynamicallypriced bids, such that the bid amount adjusts, which may be automatic,depending upon factors such as, without limitation, time of day, levelof demand, inventory availability, climate changes, competitive pricingdynamics relating to other merchants, specificity or othercharacteristic of the sender's input which generated the referral,number or rate of referrals already received or paid by the merchant, orother characteristics of sender or recipient such as any of thosementioned above (e.g., geography, user demographics, communicationchannel utilized, etc.). Embodiments of the system may provide merchantswith a dashboard which may provide, without limitation, accountsettings, preference settings, payment setting, reporting, data andanalysis, bidding controls, account management services andcommunication tools. Embodiments of the system may implement any ofthese techniques for example at least at FIG. 11, step 1112.

FIG. 8 shows an illustrative process flow for selected steps in thereferral process, for a scenario and embodiment in which input isobtained from the user via a keyboard. These steps are illustrative; oneor more embodiments may execute different steps, additional steps, ormay execute any of the steps shown in FIG. 8 in a different order or inparallel. In step 801, the system accepts input as the user provides theinput via keystrokes using a virtual keyboard connected to acommunications app. In one or more embodiments, the virtual keyboard maycontain or be linked to a referral matcher for example over any type oflocal or remote communications channel; this referral-aware keyboard mayfor example be usable with any application that accepts keyboard input,(which again may include spoken words via the microphone). Byintegrating the referral matcher with the keyboard, one or moreembodiments enables universal general-purpose referral generationcapability for any communication initiated by the user.

In step 802, the referral matcher may use artificial intelligence,natural language processing, or similar techniques to parse andunderstand the user's input. It may then select a matching item in thedatabase, and suggest this referral to the user. A matching item may beany type of information for which a referral or recommendation may berelevant, including for example, without limitation, a product, aservice, a brand, a merchant, an activity, an instruction, a suggestion,a solution, an information source, a person, and organization, aprofessional, or any combination thereof. The suggestion may beintegrated into the keyboard app, for example as a suggested wordcompletion. At step 803, the system may accept input from the user thatmay then tap on the suggested referral to accept it, which triggers step804 that creates and inserts a referral link into the communication. Thereferral link may for example contain a hyperlink to a merchant (or toan intermediary), along with a tracking code that identifies thereferring user.

Once the message containing the referral link is transmitted to therecipient, the system may execute additional steps such as steps 805through 809, for example when and if the recipient uses the referrallink to access a merchant site or to make a purchase. In step 805, ifthe recipient accesses the referral link, the link may in one or moreembodiments initially pass through an intermediate server, prior toredirecting the recipient to the merchant's site. This server may forexample track the referral, and it may place tracking information suchas cookies on the recipient's device. A cookie may for example have aduration that last for multiple days, thereby providing credit to thereferrer if the recipient transacts at another time other than theinitial click of the link. The cookie may also provide credit if therecipient subsequently goes directly to the merchant site without usingthe referral link. In cases where the recipient's device does not acceptcookies (many mobile phones do not), one or more embodiments may useother techniques such as device UID and IP address to associate arecipient's subsequent transactions with the original referral link. Inone or more embodiments, the referral link may lead the recipient to asite or user interface control that may link to one or more merchants,rather than to an intermediate server that automatically forwards to amerchant site. For example, without limitation, the referral link maylead the recipient to a jump page, an interstitial web page, a pop-up,or an overlay. This link destination may show a range of informationrelated to the referral link, including for example productdescriptions, product or merchant locations, a list of matchingmerchants from which the recipient can select, images, videos, or anyother information related to the referral.

In step 806, the system accepts input from the recipient who proceeds tothe merchant's site using the referral link. If the recipient makes apurchase or other transaction on this site, the system's referraltracker records the transaction. In step 807, a notification may be sentto the referring user by the system. In step 808, the referral trackercollects a referral credit from the merchant, which is then remitted instep 809 to the referring user. In one or more embodiments, some or allof the steps of tracking, attribution, collection, and payment may beperformed by third-party services or systems.

In one or more embodiments, determining a destination (such as ane-commerce site or a product page) for a referral may be performed intwo (or more) stages. FIG. 8A shows an illustrative process flow for anembodiment that uses two-stage matching to direct a recipient to areferral destination. Initially in step 802, as in FIG. 8, the referralmatcher determines a match for the user's input, and in step 804 thesystem generates a link with the match information. However, in one ormore embodiments the initial match in step 802 may not identify a finalreferral destination, but may instead identify only general informationsuch as the existence of a matching product, service, or merchant, or ageneral category or group of products, services, or merchants that maymatch the user's input. For example, the initial referral matching stepmay identify that a user communication mentions a flight, but it may notidentify a particular airline to fulfill this flight. In thesesituations, the initial referral link generated in step 804 may directthe recipient to an intermediate server that performs additionalprocessing and matching in step 8A05 to select a final destination forthe referral; the intermediate server may then redirect the recipient ofthe referral link to this final destination. When the recipient isredirected, the recipient arrives in step 8A06 at a final referraldestination (such as an e-commerce site), where the recipient canperform actions that result in a referral credit to the referring user(as described above with respect to FIG. 8). The final destination maybe for example, without limitation, a web site, web page, application,URL, URI, web service, or more generally any physical or virtualdestination or service that completes the referral.

A potential benefit of the process flow illustrated in FIG. 8A is thatthe initial referral matching step 802 can be relatively “lightweight”and may for example require only a relatively small database ofproducts, services, and merchants. Moreover, this database for theinitial matching may not need to be updated in real time to reflectdynamic information that may determine the best match for a referral.Because the final matching stage 8A05 is performed on the intermediateserver, it can access a more complete and up-to-date database ofinformation, including for example dynamically updated bids frommerchants on referrals. It may also access more powerful computingresources to execute computationally intensive algorithms. Theintermediate server can use any desired strategy or prioritizationalgorithms to determine which merchant to send a recipient to. Forexample, without limitation, the intermediate server may rank merchantsby any or all of price, availability, user ratings or reviews, level ofinventory, proximity to a user, speed of fulfillment, or size ofreferral credit offered to the referring user.

FIG. 8A illustrates a two-stage referral matching process flow. One ormore embodiments may generalize this process to perform referralmatching in any desired number of stages, using any desired number ofservers and databases to process information and determine a finaldestination associated with a referral link.

The process flow shown in FIG. 8 uses keyboard input as illustrativeinput into the referral process. As described above, other forms ofinput may be accepted by one or more embodiments. FIG. 9 illustratesseveral other input method options that may be supported by one or moreembodiments of the referral system. In addition to keyboard input option901 (as shown in FIG. 8), input may be obtained from a camera scanningany type of code, e.g., barcode 902, from a share button option 903,from a user selection of text or a URL 904, from a button 905 that grabsa current URL, from a camera scanning a product image 906, from imagesor audio obtained from Google™ glasses 907, from a link or button placedon any website 908, from add-ons or extensions for other keyboards 909,from an application that integrates with a physical keyboard 910, fromvoice input 911, or from type ahead, predictive word suggestions, orglide typing applications 912. These input methods are illustrative; oneor more embodiments may obtain input of any type, in any format, fromany device, service, subsystem, or application.

As a first illustrative example, which corresponds to keyboard inputoption 901, the system may be integrated into a mobile device virtual orsoft keyboard input method or service. The system may monitor inputcontent from this soft keyboard and interpret through analysis when theuser has referenced an item. When the user completes the input of areference to an item (or to an entire communication with references init), the system can offer to optimize the user's communication before itis sent. The system may do this automatically or at the user's specificrequest.

As a second illustrative example, which corresponds to voice inputoption 911, the system may be integrated into a mobile device or a smartspeaker application. The system may for example monitor words as theuser speaks them and perform analysis of the content to identifyreferences to items. Before the user sends the communication, the usermay have the system optimize the communication. This optimization may bedone automatically or upon the user's explicit request. When the systemoptimizes the communication, it augments the communication such that thereference to an item includes characteristics that include tracking andattribution (such as the user ID and the item in the database).

As a third illustrative example, which corresponds to camera inputoption 906, the system may be integrated into a camera within a mobiledevice. When a user uses a mobile device camera the system may analyzethe image in order to determine whether the image matches an item in thedatabase. If the image matches an item in the database, the system mayprompt the user to share the image in a digital communication and thesystem may transform or augment the image to include a link or to makethe image clickable (with a link embedded therein), either automaticallyor with a prompt from the user.

In one or more embodiments, a referral matcher or any related module ofthe system may analyze any type of existing content or new content thatis input by a user into any application, in order to monetize acommunication between the user and one or more recipients. The referralmatcher may be built into an application or may connect to anyapplication via any technical interface, such as for example, withoutlimitation, a local SDK, a remote API, a set of user interfacecomponents, or a back-end system to integrate any other interface orinterfaces. An application to which the system connects may be anysoftware, system, device, application, or technology that enables a userto input content or to share content with a recipient. Applications mayfor example provide communications via any digital platform, such as,without limitation, peer-to-peer communications, social media, services,messaging applications, e-commerce services, digital media, digitalcontent, images, videos, audio, product reviews, chat rooms, orpublished content. Applications may incorporate or integrate withdevices or services such as for example, without limitation, a physicalkeyboard, a virtual keyboard, a mobile device or application orsoftware, an image capture device or software, an audio/video capturedevice or software, a kiosk, a scanner, an RFID reader, a microphone, avehicle, a smart speaker, smart glasses, an augmented reality device orsoftware, a virtual reality device or software, an automated personalassistant, a smartphone, a computer, a server, a tablet, a notebook, alaptop, and any software or hardware embedded within any such device orservice or subsystem.

Content accepted by, analyzed by, or transformed by the system mayinclude content of any type or types, in any format of formats,including for example, without limitation, text, data, code,information, images, voice, video, and RFID.

A user providing input to the system may be for example, withoutlimitation, any individual, content creator, group, company,organization, system, subsystem, bot, app, application, server, orservice. The system may assign a unique user ID to each user. A user mayinteract directly with the system or may interact (either knowingly orunknowingly) via an integration of the system into an application thatthe user is using, or via a plug-in for an application or service thatthe user uses. A recipient may be for example, without limitation, oneor more individuals, companies, organizations, or entities that hear,view, read or otherwise receive the content that is input by a user orone or more links generated by the system. In one or more embodiments, auser and a recipient may be the same individual or entity.

In one or more embodiments, the system may analyze the content of acommunication using any desired methods or technologies, including forexample, without limitation, natural language processing, artificialintelligence, or image recognition. This analysis may for exampledetermine explicit references to items in the database, or implicitintent or context from which items in the database may be inferred.

As an alternative to the input method options illustrated in FIG. 9,which may for example connect the referral matcher locally with acommunications application, one or more embodiments may allow a user to“fetch” a link from a remote service. For example, a user may createcontent and send this content to a remote service, via for example atext, email, in-app communication, Facebook® post, or instant message.The system may provide or integrate with this remote service, and mayanalyze the content received to identify matches to items. The systemmay then convert matching content to links and insert the reformattedcontent with these links directly into a digital communication asspecified by the user or the system may send the reformatted contentwith these links back to the user. The user may then insert thereformatted content including the links into digital communications. Forexample, as a variation of the scenario illustrated in FIG. 7, a usermay create a message with the content “Flight to Tahoe” and send it (viaemail for example) to a remote matching service. The matching servicemay then generate the link 201 (in FIG. 7) and insert the link into adigital communication or send the link back to the user, who can theninsert the link into any desired message. When a recipient receives thelink, it functions similarly to links generated via virtual keyboards orother input methods.

FIG. 10 shows illustrative options for keyword matching or similarfunctionality that may be used to identify potential referrals withinthe user's text input or other input. Matching options may include forexample artificial intelligence 1001, natural language processing 1002,type-ahead 1003, spell-checking 1004, prompts for text when the userpresses a referral button or icon 1005, predictive typing 1006,personalization based on past typing or preferences 1007, and processingusing a recommendation engine 1008. Additional options may include forexample, without limitation, collaborative filtering, affect analysis,predictive analytics, and machine learning.

When multiple potential matches or referrals are identified, one or moreembodiments may employ business strategies and algorithms to prioritizeamong the alternatives, thereby determining which merchant, merchantsite, or coupon to present to the recipient. This prioritization may forexample analyze any factor or factors, such as for example, withoutlimitation: sender characteristics, such as the sender's itempreferences (either explicitly provided or implicitly derived) and thesender's message intent or sentiment (determined for example via NLP orhashtag analysis); recipient characteristics, such as the local timezone, location, and previous behavior including item preferences(explicitly provided or implicitly derived); item characteristics, suchas price, availability, discount amount, brand reputation, and productdelivery speed; merchant characteristics, such as conversion rate,payout amount, and reputation analysis; and bidding platformcharacteristics, such as payout amount and settlement period.

FIG. 11 shows illustrative options for data sources that may providedata for a merchant database. A merchant database may containinformation on merchants and on any products or services offered bymerchants or by any sellers. The system may aggregate items into thedatabase using any of several methods, including for example, withoutlimitation, automated fetching of content made available by merchants orother intermediaries, manual input by system administrators or users,direct self-service input by merchants, input via a bidding platform,automatic creation of new items obtained by scanning and analyzing userinput, and dynamic search of the web or other databases for matchingitems or brands that are not currently in the database. Data may beobtained directly from merchants 1101 and e-commerce sites 1102, orindirectly from content aggregators 1103. Content retrieved frommerchants or intermediaries may include for example, without limitation,product catalogs, SKUs, image files, merchant affiliations with networksor other intermediaries, promotion units (including banner ads) formerchants or specific products, and URLs with deep-links to productdetail pages for example. Data may be obtained via APIs 1104 or by usingweb crawlers/scrapers 1105. Data may be obtained from user-input anduser-generated content 1106, from a user-specified list, user favorites,or user history 1110. Databases may be network-connected 1107, embeddedwithin the software 1108, or reachable via web search 1109. Data may beobtained from aggregate listings from affiliate networks 1111. Data maybe obtained from a dynamic bidding system 1112, where merchants forexample may bid by specifying a referral fee. Data in the database maybe ranked in any desired manner, for example by user reviews/ratings,third party ratings, or user preferences 1113. Data may be obtained fromexternal databases or services indexed for example by SKUs, barcodes, orany other identifier of a product, service, or merchant. The system mayuse artificial intelligence or other methods to identify, within userinput, content such as keywords, terms describing items, or new productsand services; the system may then automatically (or via a prompt to asystem administrator) create items in the database that match the newlyidentified items. These examples are illustrative; one or moreembodiments may obtain data on merchants, products, services, referralfees, merchant locations, the availability of inventory, ratings orreviews related to merchants, products, or services, prices, or anyattributes of these items from any desired data source, in any desiredformat.

FIG. 12 illustrates a variation of the flowchart of FIG. 8, where inputis obtained from an image rather than from a keyboard. In step 1201, theuser captures an image such as a barcode, QR code, or a product image,for example using a camera integrated into or reachable via a user'sdevice, or by selecting a previously captured or otherwise accessibleimage on the user's device or accessible via a network connection. Instep 1202, the referral matcher analyzes the image to determine whichproducts, services, or merchants match the image, and it may suggestthis referral to the user. The remaining steps in this process flow maybe similar to those described with respect to FIG. 8.

FIG. 13 illustrates a variation of the flowchart of FIG. 8, where inputis obtained via a share button instead of or in addition to from akeyboard. In step 1301, the user uses a share button to share an item;the share button may for example launch a “share sheet” with sharingoptions that depend on the item being shared. The shared content is thenanalyzed in step 802, and additional steps in the process flow may besimilar to those described with respect to FIG. 8.

FIG. 14 illustrates a variation of the flowchart of FIG. 8, where inputis obtained from a user selection instead of from keystrokes of akeyboard as the user types. In step 1401, the user selects text, a URL,or any other item, within any application. In step 1402, the referralmatcher analyzes the selected information to identify one or morematching products, services, or merchants in the database; it may thenpresent the match or matches to the user as a suggestion, for example aspart of a keyboard app. In another variation, the matcher may workcontinuously in the background to match the user's input, and mayrespond with suggestions only when the user requests the matchingreferrals. The remaining steps in this process flow may be similar tothose described with respect to FIG. 8.

For the system features and capabilities described above, one or moreembodiments may perform functions that analyze system performance andoptimize the system for improved performance over time. For example,optimizations may be performed to improve utilization, utility, or valueof the system for users, recipients, merchants, or administrators.Illustrative optimizations may include for example the followingprocesses. The system may use artificial intelligence or othertechniques to observe user input content and to identify new items thatshould be added to the database. The system may aggregate and analyzedata regarding user, recipient and merchant use of the system in orderto: improve performance of links; increase fees earned; optimize whichsite or coupon is selected; and maximize conversion rates (such as thefraction of recipient links that are activated or that result intransactions). The system may aggregate and analyze data regarding thefrequency of match between content and database items (for example byevaluating ratios such as the number of item matches per word of contentor the number of item matches per communication sent); matchingalgorithms may be adjusted to increase (or decrease) these frequencies.The system may test links (periodically or continuously) to ensure thatthe URL formatting rules are functioning as intended, and in the eventof a malfunction may alert an administrator or fix the malfunctionautomatically. The system may also incorporate trust and safetyprocedures and subsystems to monitor, flag, and prohibit fraudulent useof the system, in order to protect the interests of merchants, users,recipients, and administrators.

In one or more embodiments, any combination of the functions performedby the system may utilize technology, software, resources, and servicesof third-party service providers.

In one or more embodiments, a referral link may be inserted by acommunication intermediary that conveys, transmits, routes, directs,receives, stores, forwards, resends, transforms, analyzes, combines,filters, translates, formats, edits, posts, displays, or otherwisehandles a message or other communication from a sender to a receiver.The communication intermediary may be any link, service, provider,application, server, or gateway that handles the message or any part ofthe message at any point in the path from the sender to the receiver.The intermediary may be for example a message gateway associated withthe sender, or a message gateway associated with the receiver, or amessage gateway associated with both. In some applications, the sendinguser may not need to take any action for the referral link to be addedto a message. The communication intermediary may be responsible foranalyzing the message and adding a referral link if appropriate, and theintermediary may in some situations receive credit for a completedreferral, either instead of or in addition to the sending user. In somescenarios, no software, app, or utility need be installed at all on thesending user's device or the receiving user's device, and analysis ofthe message and adding of a referral link may all occur after thecommunication has left the sender's device and before the communicationis delivered to the recipient's device. This approach may simplifymanagement of the referral process, since referral matching and referraltracking may be done centrally at one or a few communicationsintermediaries, rather than on thousands or millions of user devices. Inone or more embodiments, software operating on the sending user's orrecipient's device may communicate with the intermediary to select andadd a referral link; the sender and recipient may or may not be involvedin this process.

FIG. 15 shows an illustrative embodiment that integrates referralmatching into the message flow through one or more communicationsintermediaries. In this example, user 101 constructs a communication1500, which in this case is a text message, using mobile device 102. Inthis example, mobile device 102 has no software or application installedthat intercepts characters or text of message 1500; instead the userconstructs the entire message 1500 and sends it over normalcommunications channels. Communications carrier 1501 receives thecommunication 1500 from the device 102. Carrier 1501 may be any type ofchannel, organization, or infrastructure that conveys, transmits,receives, routes, forwards, stores, directs, edits, filters, formats, ormanages any type or types of communications. For example, withoutlimitation, carrier 1501 may be a network provider, a wireless provider,a cellular provider, an internet service provider, a social mediaplatform, a messaging service, a telephone service provider, a broadbandservice provider, a Wi-Fi provider, an email service, a text messageservice, a mobile application, a software application, a chat service,an application that conveys, transmits, routes, or formatscommunications, or a gateway between any types of devices, networks,routers, or nodes. Message 1502 may be any type or types ofcommunication, including for example, without limitation, a textmessage, an email message, a voice message, a video message, a websitelink, a link to a mobile application, a picture message, a transcribedmessage, a communication via social media, a communication via ashopping site, a communication via a message board, a posting to aproduct review service, an encrypted message, a digital communication, acomment posted to a digital media service, and a communication via anymessaging application. Carrier 1501 transmits the message 1502 to agateway 1510, which may for example process the message beforeforwarding it to the message receiver. The message 1502 may contain anidentifier 1503 of the sender, an identifier 1504 of the receiver, andthe message body 1505. The sender identifier and receiver identifier maybe for example, without limitation, phone numbers, email addresses,social media identities, device IDs or any other identifyinginformation. Messages may contain any other metadata, such astimestamps, priorities, or formats. The communication body 1505 maycontain any type or types of content, including for example, withoutlimitation, text, image, voice, audio, video, code, software, links,URLs, website addresses, and attachments.

In the illustrative embodiment of FIG. 15, gateway 1510 transmits thebody 1505 of the message to referral matcher 1511. The referral matcher1511 may execute on a computer that is coupled to or that hosts thegateway 1510. The gateway and the referral matcher may execute on thesame computer device, or on different computer devices connected by anetwork or any other type of link. In one or more embodiments thereferral matcher may execute on a computer that is coupled to both thegateway and the sender's device, or to both the gateway and thereceiver's device; the gateway, referral matcher, and sender andreceiver devices may interact in any manner to identify referral linksand transform messages. The communications intermediary that processesmessages to identify and insert referrals may be any combination ofservices executing on any combination of systems, including for examplethe sending and receiving devices and any intermediate nodes in acommunication path between the sender and receiver. A computer thatexecutes the referral matcher, the gateway, or both, may be anyprocessing device or devices, including for example, without limitation,a desktop computer, a laptop computer, a notebook computer, a server, anembedded processor, a GPU, a tablet, a phone, a wearable processingdevice, a network switch or router, or a network of any of thesedevices. In this embodiment, only the message body 1505 is transmittedto the referral matcher 1511. The sender identifier 1503 and thereceiver identifier 1504 are not transmitted to the referral matcher1511. This limited information flow to the referral matcher protects theprivacy and security of the sender and receiver. Personally identifyinginformation or sensitive personal information of the sender and receivermay be protected in that this information is not transmitted to oraccessible by the referral matcher. In one or more embodiments thegateway 1510 or the carrier 1501 may filter out other information asdesired before transmitting the message body 1505 to the referralmatcher. For example, the gateway 1510 may filter out person names (suchas “Jan” and “Cindy”), information such as bank account numbers orsocial security numbers, or any other data that may be viewed aspersonal or sensitive.

Referral matcher 1511 then analyzes the message body 1505 to determinewhether any words, phrases, text strings, or other content of the bodymatch any of the listings in database 111. This process may for examplebe similar to the matching processes described above. The referralmatcher may for example perform language processing of the message bodyto compare the content of the message body to the database listings.This language processing may for example use any techniques of naturallanguage processing, machine learning, artificial intelligence, orpattern recognition. Words may be stemmed to simplify comparison to theentries in database 111. If the referral matcher finds a match in search1512, it may then insert a link 1513 to a site or resource associatedwith the database entry. This process may be similar to the linkinsertion described above. However, in the embodiment shown in FIG. 15,the referrer in the link may be for example the carrier 1501 or anyother message intermediary or gateway, instead of the sender 1503. Thelink may be for example a hyperlink 1513 that surrounds the word orphrase in the message body that matches a listing in the database, orany other format such as for example an icon or image adjacent to thematching content. The hyperlink 1513 may link to a site or otherresource associated with the merchant, product, service, or other itemin the database that matches the message body. For example, withoutlimitation, the link 1513 may be to one or more of a website, a softwareapplication, an e-commerce service, a merchant shopping cart, a mobileapplication, a computer application, a store, a redirector, alink-tracking service, an affiliate network, a video player, a coupon orcoupon code, a promotion or promotion code, a discount code, atransaction code, a mapping service, and a URL. In one or moreembodiments the link may be to an intermediate server that redirects tothe final destination, as described above. The hyperlink 1513 may havean embedded referral code 1514 that identifies one or more referringentities, such as the carrier 1501 or the gateway 1510.

The referral matcher 1511 inserts link 1513 into the message body, andreturns the transformed message body 1506 with the link to the gateway1510. The gateway 1510 may then add the sender and receiver identifiers,and any other metadata, to the transformed message body 1506, resultingand forward the transformed message 1515 to the receiver or receivers.The message may pass through other intermediaries such as carrier 1521,which may be the same as or different from the intermediary 1501 thatoriginally carried the message from the sender's device. The receiver121 then views the message 123 with link 124 on device 122, as describedabove.

The matching process 1512 of the referral matcher 1511 may not identifya match between a message body and the database 111. In this situationthe referral matcher may return a no-match result 1513 to the gateway,which indicates that the gateway should forward the original message tothe receiver.

In one or more embodiments, a referral link may include both theidentity of a communication intermediary and an identity of the sender.This situation is illustrated in FIG. 15A. As in FIG. 15, message 1502is transmitted to gateway 1510, which forwards message body 1505 toreferral matcher 1512. In this embodiment, the sender identifier 1503 isencoded in process 1530, and an encoded sender identity 1531 is alsopassed to the referral matcher. The referral matcher 1511 then insertsan embedded referral tracking code 1514 for the carrier (or otherintermediary) and another referral tracking code 1514 a with the encodedsender identity. Because the sender identity is encoded, the privacy andpersonal information of the sender is preserved; however, the sender maybe credited for the referral using the encoded identifier. One or moreembodiments may put any desired information into one or more embeddedreferral tracking codes that are integrated into a referral link; thisinformation may identify either directly or anonymously via encoded dataany party involved in the creation or transmission of a communication.

In one or more embodiments, a communications intermediary that processesa message may be associated with the receiver of a message rather than,or in addition to, the sender. This scenario is illustrated in FIG. 15B,which is a variation on the embodiment shown in FIG. 15A. Message 1500is transmitted through two communications intermediaries: first throughcarrier 1501 that is associated with the sender, and then throughcarrier 1521 that is associated with the receiver. Other communicationsintermediaries may exist between carriers 1501 and 1521. Messaginggateway 1510 is linked to carrier 1501; however, unlike the scenario inFIG. 15, this gateway 1510 does not transform the message to insert areferral link. Instead the message is forwarded to carrier 1521 and asecond message gateway 1510 b that is associated with the receiver'scarrier 1521. The receiver gateway 1510 b forwards the body 1505 of themessage to the referral matcher, which inserts the referral link andreturns the transformed message body 1506 b to the gateway 1510 b. Thetransformed message 1515 b is then sent to the receiver's device 122. Inthis scenario, the referral link 124 b may be associated with thereceiving gateway 1510 b or the receiving communications carrier 1521,for example.

In one or more embodiments, the referral link may contain the identityof the receiver, instead of or in addition to the identity of thecommunications intermediary associated with the receiver, or in additionto identities of any other intermediaries or of the sender. Thisreceiver identity may be encoded, similarly to the encoding of thesender identity illustrated in FIG. 15A. In the example shown in FIG.15B, the transformed link 124 b in message body 1506 b may for examplecontain the identity 1514 b of the communications intermediary 1521, andthe encoded identity 1516 b of the receiver 121 or of the receivingdevice 1222. Referral links may in general contain identities (unencodedor encoded) of any of the parties or systems involved in acommunication, including the sender, receiver, carriers, gateways, andother systems involved in message transmission, storage, transformation,or delivery.

Instead of or in addition to generating a link based on references todatabase entries found in messages, one or more embodiments maytransform messages in the opposite sense, and may recognize databaseentries based on links a user has placed messages. FIG. 15C shows anillustrative example of this process. User 101 initially views screen1541 on mobile device 102, which may be for example a web browserscreen. The user views a web page or similar resource with URL or otheridentifier 1543, and decides to send this link to one or more recipientsin a message. User 101 therefore performs a copy and paste 1544 andinserts the link into message 1500 c in screen 1542, which may be forexample a messaging app. The message is sent via carrier 1501 andgateway 1510, as described with respect to FIG. 15, and the messagecontent 1505 c is transmitted to the referral matcher 1511. In thisembodiment, database 111 includes or references a table or other datastructure 1545 that associates URLs or similar links with databaseentries. The referral matcher therefore identifies that the URL 1543 inmessage body 1505 c corresponds to entry 1546. As a result of thismatch, it makes the transformation 1513 c to convert the reference to ahyperlink that identifies the entry 1546 and credits the communicationsintermediary with an embedded referral code 1514 c. One or moreembodiments may in addition or instead insert an embedded referral codethat references the sender, receiver, or any other entity.

In one or more embodiments, the transformed communication may include ormay consist of instructions for modifications to the message that may beexecuted at a later stage in the path of the message to the receiver.FIG. 15D shows an illustrative embodiment with a variation on theprocess flow of FIG. 15, where the initial message transformationgenerates transformation instructions to add a link to the message, andthe message body is modified afterwards based on those transformationinstructions. As in FIG. 15, message 1500 is transmitted tocommunications intermediary 1501, which forwards it to gateway 1510. Thebody 1505 of the message is processed by referral matcher 1511, whichfinds a match and determines that a replacement 1513 of a term by a linkis indicated. However, the referral matcher in this embodiment does notmake the replacement directly; instead it returns transformationinstructions 1513 a to the gateway 1510. The gateway sends theseinstructions 1513 a to the communications intermediary 1501, whichexecutes the transformation instructions to generate the transformedmessage 1506. In one or more embodiments, the replacement oraugmentation of a portion of a message with a link may be performed atany stage in the transmission of the message from the sender to thereceiver; the referral matcher may execute this replacement oraugmentation, or it may provide instructions that any later stage orstages may perform to effect the replacement or augmentation of themessage.

FIG. 16 continues the example of FIG. 15 to show the referral link beingused by a receiver of a communication. If receiver 121 clicks on thelink 124, the link directs to site 201 a, with an embedded referraltracking code in the URL. If the receiver then performs an action suchas a purchase by pressing the button 203, the merchant 212 may transmita notification 1601 of the transaction to referral tracker 1602. Thereferral tracker 1602 may be integrated with the referral matcher, or itmay be a separate system. The notification 1601 may include the referraltracking code, so that the referral tracker can calculate a credit 211 aand record or transmit this credit to the appropriate entity 1501, whichin this example is the carrier. For the scenario illustrated in FIG.15A, credits may be provided to both a communication intermediary and tothe sender of the message. For the scenario illustrated in FIG. 15B,credits may be provided to both the communications intermediaryassociated with the receiver, and to the receiver of the message.

In one or more embodiments, the referral matcher 1511 may performadditional analysis of the body of a communication to determine whetherthe context of the match indicates that adding a referral link iswarranted. FIG. 17 shows an example where message 1500 d mentions amerchant, but in a negative sense. The referral matcher 1511 may firstperform matching search 1512 on message body 1505 d to determine thatthe merchant is mentioned or otherwise matches the communication body,and then perform sentiment analysis 1701 to determine whether themention is positive, neutral, or negative for example. Sentimentanalysis 1701 may use any of the techniques known in the art todetermine the sentiment associated with a message. For example,sentiment analysis 1701 may track the usage of positive or negativewords, particularly if they are in proximity to the located matchingwork or phrase. In this example, sentiment analysis determines that thesentiment 1702 is negative; therefore, the referral matcher returns anindication 1703 to the gateway that it has not added a referral link.

In one or more embodiments, the referral matcher may analyze the body ofa communication to determine whether a database entry is referenced inor with a context that indicates that a referral link is appropriate.The required context may for example include certain trigger words ortrigger phrases that must appear in proximity to the mention of the itemin order to generate a referral link. This situation is illustrated inFIG. 17A. Referral database 111 contains or references a set of triggerwords or phrases 1711 associated with entries in the database or withcategories of entries. Message 1500 e is sent from device 102, andcorresponding message body is transmitted to referral matcher 1511. Thematching step 1512 indicates that an item from database 111 occurs inthe message body. The subsequent step 1710 searches the message body1505 e for additional context that may be required to generate areferral. For the matched item, the trigger word 1712 from table 1711appears in the message body 1505 e in a position that is near thematched item; therefore, the referral link is inserted into transformedmessage body 1506. If no trigger word or phrase were located, then theexistence of the positive match from the initial matching step 1512 maynot be sufficient to generate a referral link in this illustrativeexample.

In one or more embodiments, the closeness of a match needed between thetext of a communication body and a listing in the database may be aconfigurable parameter that may be set for example by the gateway or bya communication intermediary. FIG. 18 shows an illustrative embodimentwith three levels of “closeness of match” defined. These levels areillustrated in table 1805 for entry 1810 of database 111. The mostspecific type of match is “exact,” which requires that specific words orphrases describing the particular merchant, brand, or similar item bepresent in the body of a communication. The next, less specific type ofmatch is “category,” which is a grouping of a generic class of productsor services into which the associated listing belongs. The leastspecific type of match is “activity,” which describes activities forwhich the associated listing is related. These groupings areillustrative; one or more embodiments may use any type of hierarchy orother organization of matching levels. The gateway 1510 (or otherentity) may set a closeness-of-match parameter 1801 from among thesethree levels to control how the referral matcher 1511 operates. In theexample of FIG. 18, if parameter 1801 is value 1803 a (exact), then thematcher indicates no match 1804 a, since the words associated with exactmatch do not appear in the message; however if parameter 1801 is either1803 b (category) or 1803 c (activity), then the referral matcher doesfind a positive match. For message 1500 f with message body 1505 f, thereferral matcher may for example insert a referral link to listing 1810around the category word “flight” in transformed message body 1506 f.

Particularly for non-exact matches, a referral matcher may in somescenarios identify multiple listings in database 111 that match acommunication body, even for the same word or phrase. In thesesituations, the referral matcher may select from among the multiplematches, using for example prioritization processes as described abovebased on factors such as conversion rates, payout rates, or sellerreliability. In some situations, a listing in database 111 may beassociated with multiple sites, and generation of a referral link mayinclude selection of a specific site from these multiple sites. FIG. 18Ashows an illustrative embodiment with listing 1546 in database 111associated with multiple sites 1820. In this example, listing 1546 is abrand, and the sites 1820 associated with the list correspond toretailers that offer this brand for purchase. One or more embodimentsmay associate multiple sites with a database listing in any desiredmanner. Referral matcher 1511 may select a specific site from the sites1820, and may insert a link to that selected site into the transformedcommunication. This selection may be based on any criterion or criteria,as described above. In the example shown in FIG. 18A, data 1821associated with each site is available to the referral matcher (eitherin database 111 or as another data source accessible to the referralmatcher). This data shows the amount of referral fee each retailer paysfor a completed transaction resulting from a recipient clicking on areferral link and completing a purchase. In this example, the site 1822with the highest payment per sale is selected and is inserted as link1823 into transformed message 1824. Any performance metric in additionto or instead of payment data may be used to select a specific site frommultiple sites associated with a listing in one or more embodiments.Illustrative performance metrics associated with a site that may be usedin one or more embodiments include for example, without limitation, theamount of referral credit provided by the site, the price of items onthe site, the closeness of match of the site to the communication, thetransaction conversion rate of the site, a defined set of business logicassociated with the site, closeness of match of the site tocharacteristics of the sender or receiver, proximity of the location ofthe site to the sender location or receiver location, speed offulfillment by the site, and a review score, popularity score, or ratingscore associated with the site.

In one or more embodiments, the selection of a specific site frommultiple matching sites may be performed after a recipient clicks on areferral link, as described for example above with respect to FIG. 8A.The referral link may send a recipient to an intermediate server, whichthen accesses data such as a list of possible destination sites 1820 andmetrics 1821 associated with each site in order to select a finaldestination site.

In one or more embodiments, language processing or other analysis of thebody of a communication may allow the referral matcher to generate amore specific link that incorporates additional information from thecommunication. FIG. 19 shows an illustrative embodiment in whichlistings in the database 111 may be associated with subcategoryinformation that may be used to identify a more specific page on a siteor specific parameters in a referral link URL. The type of subcategoryinformation that is relevant may in general vary among listings in thedatabase 111. For example, for clothing brands, relevant subcategoryinformation may include data such as size, style, or color; for airtravel, on the other hand, relevant subcategory information may includedata such as departure, destination, date, and time. In the exampleshown in FIG. 19, referral matcher 1511 first performs a search 1901 fora primary target match between the body of message 1500 g and listingsin database 111. The result is a category match 1902 that is associatedwith listing 1903. Database 111 contains a table 1910 of subcategoryinformation linked to this listing, which indicates that location anddate/time information may be relevant to a referral link for this entry.This data is used in a subsequent search 1904 for modifiers in themessage body that may indicate the subcategories of flights that arerelevant. This search may look for certain keywords or categories ofwords or phrases, such as those identified in table 1910, or it may useany other type of natural language processing or artificial intelligenceto infer modifiers and context. The search results 1905 may be used inlink generation process 1906, which generates referral link 1907. Thislink 1907 passes the location data and date/time data (which the systemmay convert from the generic “next weekend” to a specific date) to thetarget site as URL parameters. One or more embodiments may usesubcategory data to refine a referral link in any desired manner,including for example linking to a specific page on a site, linking toone of several related sites, or passing data to a web page in URLparameters.

One or more embodiments may analyze message histories in addition toindividual messages to generate referral links. For example, the contentof message 1500 g of FIG. 19 may be spread over two or more messages,and the system may analyze the combined content of these messages todetermine whether a referral link should be added and possibly todetermine the specific target of the link. For instance, a first messagemay contain the first sentence of message 1500 g, which indicates thedestination. A subsequent message may contain the remainder of message1500 g, which indicates the date. The system may track the messagehistory to generate the specific referral link 1907, which combines thelocation data from a previous message with the date information from asubsequent message.

In one or more embodiments, senders or receivers of messages may be ableto opt-out of the link insertion process performed by a communicationintermediary or gateway. This feature is illustrated in FIG. 20, whichillustrates a variation on the example of FIG. 15. In this example, thegateway 1510 passes the message body 1505 to referral matcher 1511 andreceives a transformed body 1506 with an inserted referral link. Thegateway (or the carrier 1501) recognizes that this message is the firstmessage from the sender 101 for which a referral link has beengenerated. Therefore, it generates a message 2001 to the sender thatinforms the sender that the sender's message has been automaticallyconverted to a “smart message” with an embedded link. The sender has theoption of learning more about the process, and of opting out of havinglinks added to messages from this sender. If the sender chooses to optout, the gateway 1510 or the carrier 1501 may track this information andmay refrain from sending future messages from this sender to thereferral matcher 1511. One or more embodiments may provide a similaropt-out capability for the receiver. One or more embodiments may insteadrequire the sender or receiver to opt in to the referral link processbefore generating these links. In addition to or instead of opting in oropting out, in one or more embodiments the sender or receiver may beable to send any other types of communication preferences messages tothe referral matcher or to any of the communications intermediaries.Communications preferences messages may for example modify the frequencywith which referral links are inserted, or the manner in which thereferral links are inserted, displayed, or used. For example, arecipient may request that referral links be inserted only from selectedsenders, only for selected topics, or only at specific times. Arecipient may request that referral links be highlighted in messages asreferral links, to distinguish them from other links that may already bein messages, or that referral links be displayed only at certain timesor under certain conditions. A sender may similarly request for examplethat referral links be inserted only for selected recipients, only at acertain frequency, or only for certain topics.

One or more embodiments may process images, and may generate referrallinks based on analysis of these images. Images may be obtained forexample from any digital communication, such as an email, text message,or social media post. As described above, processing of images andgeneration of referral links may occur at any node or nodes in acommunication link between a sender and a receiver, including forexample the sender's device, the receiver's device, a messaging gateway,a communications server, or any combination thereof. Processing mayoccur on any device used to encrypt or decrypt all or a portion of acommunication, including for example a device used by the sender toencrypt a communication or a device used by the receiver to decrypt acommunication.

FIG. 21 shows a flowchart of illustrative steps that may be used in oneor more embodiments to generate referral links from images. One or moreembodiments may perform only a subset of these steps, for exampleomitting any of the steps in FIG. 21, and may reorder and restructuresteps in any desired manner. Any of these steps shown in FIG. 21 may beperformed by one or more third-party services or systems. In step 2101,an image input is obtained, for example from a digital communication orfrom a device or image library. Illustrative sources of images mayinclude for example, without limitation, a camera, an email, a textmessage, a picture message, a video message, a Snapchat™ message, aFacebook™ message, a Twitter® message, a post on any website or blog, ora gallery from a photo sharing or photo archiving site. Images may alsobe obtained from videos associated with any of these sources. Images maybe for example, without limitation, photos, videos, video frames,drawings, paintings, graphics, 2D models, 3D models, screenshots,barcodes, QR codes, icons, emoticons, avatars, or virtual reality scenesor characters. In step 2102, the components of an image are detected andclassified. Components 2120 may include for example, without limitation,any or all of text, object, scene, code (such as barcodes or QR codes orsimilar identifying marks), product, product packaging, label, logo,building, place, location, person, character, celebrity, performer,activity, event, shapes, symbol, action, colors, icon, emoji, andsticker. A single image may have one or more types of components. Step2103 then routes individual components to analyzers that may for examplebe specialized to analyze a specific type or types of component. Foreach component type there may be a distinct extraction method. Therouting methodology may split the image into its components and mayroute each component to a corresponding analysis method. In one or moreembodiments, different component analyzers may execute on differentsystems, and routing may be performed over network connections. Step2104 then analyzes the components of the input image, using for exampleindividual analyzers such as a text analyzer, a logo analyzer, an iconanalyzer, etc. Depending on the component type, extraction of thecomponent from the image and analysis of the extracted component mayinclude various types of processing. For example, for text components,analysis may recognize the letters, words, or phrases in a component.Text analysis may process or transform the orientation, font size, fonttype, boundary, decoration, and color; it may de-noise and de-warp theextracted component. For image components, analysis may identify theobject or objects contained in an image component. Image processing mayinclude for example de-noising, de-warping, boundary detection, logoanalysis, analysis of orientation, shape, and color, and reverse imagelookup. For codes (such as barcodes or QR codes), the codes may beextracted and decoded, or they may bypass extraction and skip to thesynthesis or matching steps described below.

Results of the component analyzers from analysis step or steps 2104 maythen be transmitted to step 2105 that synthesizes the various analyses.The synthesis stage 2105 may for example determine relationships amongthe extracted component data, and reconstruct the extracted and analyzedcomponents of the images. The extracted component data may besynthesized into a text descriptor of the image, which may include forexample a list of keywords or key phrases, a hierarchically structureddescriptor of the image components, or a summary sentence or phrase thatdescribes the image and its components. The result of synthesis 2105 isthen transmitted to referral matching step 2106, which searches referraldatabase 111 for one or more matches, as described above. After matchingagainst an item in the referral database, an optional verification step2107 may verify the match or matches, for example by comparing theoriginal image input to an image of the item that it has matched againstin the referral database 111. This verification step 2107 may beoptional in one or more embodiments. It may not be performed at all inone or more embodiments, and in one or more embodiments it may beperformed only in certain situations. Verify match step 2107 may forexample generate a confidence score based on how closely the originalinput image matches an image of the matched item or items. Thisconfidence score may provide a measure of how close the match is. Inembodiments that perform optional verification step 2107, the system mayset a threshold for the confidence score, and may proceed withgenerating a referral link for the input image only if the confidencescore is above the threshold. The confidence score and threshold may notbe used in embodiments that do not perform optional verification step2107.

If matching step 2106 generates multiple matches against referraldatabase 111, then a prioritization step 2108 may be performed to selectone or more of these matches for referral links. This prioritization mayuse any information from the component analyses 2104 and any informationfrom the referral database 111 to prioritize matches. For example, asdescribed above, merchants may be prioritized based on factors such asthe size or amount of the referral credit associated with each merchant,the location of the merchant, the availability of inventory, speed offulfillment of orders, ratings or reviews related to the merchant, orthe price of the product or service offered by the merchant. Ifverification step 2107 is performed, then the confidence score generatedfor each match may also be used for prioritization; for example, thematch with the top confidence score may be selected, or all matches witha confidence score above a threshold may be selected. One or moreembodiments may use any desired method to prioritize matches and toselect which match or matches to use for referral links.

Step 2109 then generates one or more referral links, as described above.Each referral link is trackable, and may include for example theidentity of one or more of the matched item, the sender, the receiver, amessaging intermediary, or a gateway. In step 2110, the referral link orlinks are presented by modifying or augmenting a message orcommunication that is displayed to a recipient. For example,presentation step 2110 may construct and transmit markup instructions towhatever system or systems are responsible for formatting or displayinga message. Formatting transformations may occur for example at thesending device, at the receiving device, at the messaging gateway, at amessaging intermediary, at a communications server, in any applicationor software, or using any combinations thereof. The message body may beformatted in one or more ways to incorporate the trackable referrallink. For example, the image in the message may be made clickable orinteractive, or one or more elements may be placed adjacent to or overthe image. Added elements may include for example a URL, a link, atoken, a sticker, a button, an image, a product name, or a productdescription. One or more embodiments may present a referral link bytransmitting an additional message with any of the above elementsincluded.

FIG. 22 shows selected processing steps applied to an illustrative image2200. This image 2200 may for example be sent from a sender to areceiver in a message, or shared between a sender and a receiver. Instep 2102, individual components of the image are detected andclassified. For example, this step may identify a text component 2201,an object 2202, and a QR code 2203 attached to an item in the image.Analysis step 2104, which may occur on various systems using analyzersappropriate for each component type. Each analysis may for examplegenerate a list of keywords or phrases associated with the components.Text 2211 may be for example a direct translation of the textual contentof item 2201 in the image. Keyword 2212 may be the result of applying anobject recognizer (such as a neural network) to object 2202. Descriptor2213 may result from decoding QR code 2203. In step 2105, the individualcomponent analyses are synthesized into a descriptor 2220 of the entireimage. This descriptor may include for example a list 2220 a of keywordsand phrases; in one or more embodiments it may be structuredhierarchically to match for example a hierarchical decomposition ofimage components. The descriptor 2220 may include for example a summarysentence or phrase 2220 b that describes the image and its majorcomponents. Step 2106 then matches descriptor 2220 against the referraldatabase, generating a match against item 2221, such as a merchant.Verification step 2107 may for example retrieve an image 2222 associatedwith the matched item 2221, and compare this image 2222 to the originalinput image 2200. A match score 2223 may indicate how closely the twoimages match. In this illustrative scenario, the two images are similarbecause they contain some similar elements, but they are not closematches. Depending on the threshold set for the match score 2233, areferral link 2224 may be generated and added to image 2200 for displayto the image recipient.

FIG. 23 shows a variation of the example of FIG. 22 where the matchreferral database step 2106 generates multiple matches for thecomponents of the image or the image descriptor 2220. In this example,component 2201 generates matches 2311 in category 2301 of booking agentsthat may for example be able to reserve stays at the identified bed andbreakfast 2211; component 2202 generates matches 2312 in category 2302of sellers of the identified item 2212 (a bicycle); and component 2203generates matches 2313 in category 2303 of rental services similar to oridentical to the identified service 2213. The multiple matches 2311,2312, and 2313 are then input into prioritization step 2108, whichselects one or more matches from which to generate referral links.Prioritization and selection may be based for example on closeness ofmatch, or on any characteristics of the matched merchant or service, asdescribed above. In this example, two matches 2311 c and 2313 b areselected in step 2108, and these matches are input into step 2109 forgeneration of referral links, which are transmitted to step 2110 thatpresents the referral links to the receiver.

While the invention herein disclosed has been described by means ofspecific embodiments and applications thereof, numerous modificationsand variations could be made thereto by those skilled in the art withoutdeparting from the scope of the invention set forth in the claims.

What is claimed is:
 1. A monetization system for images comprising: adatabase comprising one or more listings, each listing of said one ormore listings comprising one or more of a product, a service, a brand, amerchant, a name of a merchant, a name of a web site, a name of aproduct, a name of a service, a location, a review, a rating, a productnumber, a model number, a description, a picture, an image, a diagram, abarcode, a UPC number, an RF code, an activity, a keyword, a phrase, aproduct category, an SKU, an instruction, a suggestion, a solution, aninformation source, a person, an organization, a professional; acomputer coupled to said database in a communication path between afirst user and a second user, wherein said computer receives acommunication between said first user and said second user; saidcommunication comprises one or more of an identifier of said first user,an identifier of said second user, a communication body including afirst image; said computer is configured to execute specificinstructions to implement a referral matcher that is configured toprocess said first image to determine whether a positive match existsbetween said first image and at least one of said one or more listings;when said positive match exists, generate a referral link that comprises a link to a site associated with said at least one of said one or morelistings; and  an embedded referral tracking code that identifies areferrer, wherein said referrer comprises one  or more of  said firstuser,  said second user,  a communication intermediary in saidcommunication path between said first user and said second user;transform said communication to a transformed communication containingsaid referral link; and transmit said transformed communication alongsaid communication path to said second user; and, implement a referraltracker that is configured to obtain a notification when said seconduser uses said referral link and performs an associated task; and,transmit or record a referral credit to said referrer based on saidnotification.
 2. The system of claim 1, wherein said computer comprisesone or more of a device used by said first user; a device used by saidfirst user to encrypt at least a portion of said communication; a deviceused by said second user; a device used by said second user to decryptsaid at least said portion of said communication; a device used by saidcommunication intermediary.
 3. The system of claim 1, wherein saidcommunication intermediary comprises one or more of a communicationscarrier; a network provider; an internet service provider; a socialmedia platform; a messaging service; a telephone service provider; abroadband service provider; a Wi-Fi provider; an email service; a textmessaging service; a mobile application; a software application; a chatservice; an application which conveys, transmits, receives, routes, orformats communications; a gateway coupled to or executing on saidcomputer.
 4. The system of claim 1, wherein said communication comprisesone or more of a message, a text message, an email message, a voicemessage, a video message, a website link, a link to a mobileapplication, a picture message, a transcribed message, a communicationvia social media, a communication via a shopping site, a communicationvia a message board, a posting to a product review service, an encryptedmessage, a digital communication, a comment posted to a digital mediaservice, a communication via a messaging application.
 5. The system ofclaim 1, wherein said uses said referral link comprises performs one ormore of a tap, click, gesture, response, user interface interaction,verbal command.
 6. The system of claim 1, wherein said associated taskcomprises one or more of a click, view, visit, transaction, purchase,reservation, subscription, sign-up, submission, software installation,download, inquiry, content consumption, survey completion, participationin a digital interaction.
 7. The system of claim 1, wherein said sitecomprises one or more of a website, a software application, ane-commerce service, a merchant shopping cart, a mobile application, acomputer application, a store, a redirector, a link-tracking service, anaffiliate network, a video player, a coupon or coupon code, a promotionor promotion code, a discount code, a transaction code, a mappingservice, a URL.
 8. The system of claim 1, wherein said database furthercomprises one or more of a file, library, catalog, directory, opengraph, real-time web search, cached web search result, data feed.
 9. Thesystem of claim 1, wherein said process said first image to determinewhether said positive match exists comprises detect one or morecomponents in said first image and determine a component type for eachcomponent of said one or more components; generate component analyses ofsaid one or more components; synthesize said component analyses to forma descriptor of said first image; and, determine whether said descriptorcorresponds to said at least one of said one or more listings.
 10. Thesystem of claim 9, wherein said computer is further configured toretrieve a second image corresponding to said at least one of said oneor more listings; calculate a confidence score based on how closely saidsecond image matches said first image; and, generate said referral linkwhen said confidence score exceeds a threshold value.
 11. The system ofclaim 1, wherein said computer is further configured to execute specificinstructions to receive or access a closeness-of-match parameter; and,said process said first image to determine whether said positive matchexists further comprises compare said closeness-of-match parameter tohow closely said first image corresponds to said at least one of saidone or more listings.
 12. The system of claim 1, wherein said processsaid first image to determine whether said positive match exists furthercomprises determine whether said first image matches a subcategorywithin said at least one of said one or more listings; when said firstimage matches said subcategory, generate said referral link comprisingone or both of a link to a page in said site that corresponds to saidsubcategory; and, a link comprising one or more URL parameters thatcorrespond to said subcategory.
 13. The system of claim 12, wherein saidsubcategory comprises one or more of a size, a style, a model, a type, asub-brand, a feature, a characteristic, a quality, a name, a stockkeeping unit, a color, a date, a time, a location, a quantity.
 14. Thesystem of claim 1, wherein one or both of said computer and saidcommunication intermediary are configured to receive a communicationpreferences message from one or both of said first user and said seconduser; when said communication preferences message is received, modifysaid referral matcher to adjust one or both of a frequency or a mannerwith which referral links are added or displayed in messages from saidfirst user to said second user.
 15. The system of claim 1, wherein saidreferral matcher is further configured to select said site associatedwith said at least one of said one or more listings from two or moresites when said positive match exists and when said two or more sitescorrespond to said at least one of said one or more listings.
 16. Thesystem of claim 15, wherein said select said site associated with saidat least one of said one or more listings from said two or more sitescomprises compare said two or more sites based upon a performancemetric; and, select a best site with a best performance metric as saidsite associated with said at least one of said one or more listings. 17.The system of claim 16, wherein said performance metric associated witha specific site comprises one or more of an amount of referral creditassociated with said specific site; a price associated with saidspecific site; how closely said first image corresponds to said specificsite; a transaction conversion rate associated with said specific site;a defined set of business logic associated with said specific site; howclosely characteristics of said first user correspond to said specificsite; how closely characteristics of said second user correspond to saidspecific site; proximity of a location of said first user to saidspecific site; proximity of a location of said second user to saidspecific site; a location associated with said specific site; a speed offulfillment associated with said specific site; a review score, apopularity score or a rating score associated with said specific site.18. The system of claim 1, wherein said site comprises an intermediateserver; and, when said second user accesses said site via said referrallink, said intermediate server is configured to identify one or moredestinations that match said referral link; select a specificdestination from said one or more destinations; redirect said seconduser to said specific destination.
 19. The system of claim 18, whereinsaid select said specific destination from said one or more destinationsthat match said referral link comprises compare said one or moredestinations on a performance metric; and, select a destination fromsaid one or more destinations with a best performance metric.
 20. Thesystem of claim 19, wherein said performance metric associated with adestination comprises one or more of an amount of referral creditassociated with said destination; a price associated with saiddestination; how closely said image corresponds to said destination; atransaction conversion rate associated with said destination; a definedset of business logic associated with said destination; how closelycharacteristics of said first user correspond to said destination; howclosely characteristics of said second user correspond to saiddestination; proximity of a location of said first user to saiddestination; proximity of a location of said second user to saiddestination; a location associated with said destination; a speed offulfillment associated with said destination; a review score, apopularity score or a rating score associated with said destination.